文稿機器學習classification

123456789Givenacollectionofrecords(trainingsetEachrecordcontainsasetofattributes,oneoftheattributesistheclass.Findamodelforclassattributeasafunctionofthevaluesofotherattributes.Goal:previouslyunseenrecordsshouldbeassignedaclassasaccuratelyaspossible.Atestsetisusedtodeterminetheaccuracyofthemodel.Usually,thegivendatasetisdividedintotrainingandtestsets,withtrainingsetusedtobuildthemodelandtestsetusedtovalidateit.IllustratingClassification123456789Training?????TestExamplesofClassificationPredictingtumorcellsasbenignorClassifyingcreditcardtransactionsaslegitimateorfraudulentClassifyingsecondarystructuresofproteinasalpha-helix,beta-sheet,orrandomCategorizingnewsstoriesasweather,entertainment,sports,DecisionTreebasedRule-basedMemorybasedNeuralNa?veBayesandBayesianBeliefSupportVectorExampleofaDecision1123456789< Model:DecisionAnotherExampleofDecision123456789<Therecouldbemorethanonetreethatfitsthesamedata!DecisionTreeClassification12345678967K?Training T???? Apply ApplyModeltoTestStartfromtherootof

?>>ApplyModeltoTest?>>ApplyModeltoTest?>>ApplyModeltoTest

?>>ApplyModeltoTest?>>ApplyModeltoTest? to>>DecisionTreeClassification123456789??????D DecisionTreeManyHunt’sAlgorithm(oneoftheID3,GeneralStructureofHunt’s12123456789thatreachanodeGeneraltIfDtcontainsrecordsthatbelongthesameclassy,thentisaleafnodelabeledasyttIfDtisanemptyset,thentisaleafnodelabeledbythedefaultclass,ydIfDtcontainsrecords ?belongtomorethanone?useanattributetesttosplitthedataintosmallersubsets.Recursivelyapplytheproceduretoeachsubset.123456789

< >=TreeGreedySplittherecordsbasedonanattributetestthatoptimizescertaincriterion.DeterminehowtosplittheHowtospecifytheattributetest（二元屬性、標稱屬性、序數(shù)屬性、連續(xù)屬性HowtodeterminethebestDeterminewhentostopTreeGreedySplittherecordsbasedonanattributetestthatoptimizescertaincriterion.DeterminehowtosplittheHowtospecifytheattributetestHowtodeterminethebestDeterminewhentostopHowtoSpecifyTestDependsonattributeDependsonnumberofwaysto2-waysplitSplittingBasedonNominalMulti-waysplit:Useasmanypartitionsasdistinct

Binarysplit:DividesvaluesintotwoNeedtofindoptimal

SplittingBasedonOrdinalMulti-waysplit:UseasmanypartitionsasdistinctBinarysplit:DividesvaluesintotwoNeedtofindoptimal

Whataboutthis SplittingBasedonContinuousDifferentwaysofDiscretizationtoformanordinalcategoricalStatic–discretizeonceattheDynamic–rangescanbefoundbyequalintervalbucketing,equalfrequencybucketing(percentiles),orclustering.BinaryDecision:(A<v)or(A?considerallpossiblesplitsandfindsthebestcanbemorecompute>SplittingBased><> (i)Binary (ii)Multi-wayTreeGreedySplittherecordsbasedonanattributetestthatoptimizescertaincriterion.DeterminehowtosplittheHowtospecifytheattributetestHowtodeterminethebestDeterminewhentostopHowtodeterminetheBestBeforeSplitting:10recordsofclass10recordsofclass WhichtestconditionistheHowtodeterminetheBestGreedyNodeswithhomogeneousclassdistributionarepreferredNeedameasureofnodeC0:C0:C1:C0:C1:Highdegreeofimpurity

LowdegreeofMeasuresofNodeGiniHowtoFindtheBestBefore

Node Node NodeNodeNodeNodeNode Gain=M0–M12vsM0–MeasureofImpurity:GiniIndexforagivennodetGiniGini(t)=1-[p(j|(NOTE:p(j|t)istherelativefrequencyofclassjatnodeMaximum(1-1/nc)whenrecordsareequallydistributedamongallclasses,implyingleastinterestinginformationMinimum(0.0)whenallrecordsbelongtooneclass,implyingmostinterestinginformation06 06152433ExamplesforcomputingGiniGini(t)=1-[p(j|P(C1)=0/6= P(C2)=6/6=06Gini=1–P(C1)2–P(C2)2=1–0–10615P(C1)=1/6 P(C2)=5/6Gini=1–(1/6)2–(5/6)21524P(C1)= P(C2)24Gini=1–(2/6)2–(4/6)2=SplittingBasedonUsedinCART,SLIQ,Whenanodepissplitintokpartitions(children),thequalityofsplitiscomputedas, ni=numberofrecordsatchildi,n=numberofrecordsatnodep.BinaryAttributes:ComputingGINISplitsintotwoEffectofWeighing–LargerandPurerPartitionsaresought=1–(5/7)2–=

=1–(1/5)2–=NodeNode

NodeC1NodeC166Gini=5124=7/120.4085/12*=CategoricalAttributes:ComputingGiniForeachdistinctvalue,gathercountsforeachclassinthedatasetUsethecountmatrixtomake

(findbestpartitionof1214113124221512123456789SeveralChoicesforthesplittingvalue Numberofpossiblesplittingvalues=NumberofdistinctEachsplittingvaluehasacountmatrixassociatedwithitClasscountsineachofthepartitions,A<vandA?vSimplemethodtochoosebest>Foreachv,scanthedatabase>gathercountmatrixandcomputeitsGiniindexRepetitionofContinuousAttributes:ComputingGiniForefficientcomputation:foreachSorttheattributeonLinearlyscanthesevalues,eachtimeupdatingthecountmatrixandcomputingginiindexsTaxablesTaxable>>>>>>>>>>>03030303122130303030300716253434343443526170SplitPositionAlternativeSplittingCriteriabasedonEntropyatagivennodeEntropyEntropy(t)=-p(j|t)logp(j|(NOTE:p(j|t)istherelativefrequencyofclassjatnodeMeasureshomogeneityofaMaximum(lognc)whenrecordsareequallydistributedamongallclassesimplyingleastinformationMinimum(0.0)whenallrecordsbelongtooneclass,implyingmostinformationEntropybasedcomputationsaresimilartoGINIindexExamplesforcomputingEntropyEntropy(t)=-p(j|t) p(j|j206P(C1)=0/6= P(C2)=6/606Entropy=–0log0–1log1=–0–0=15P(C1)= P(C2)15Entropy=–(1/6)log2(1/6)–(5/6)log2(1/6)=24P(C1)= P(C2)24Entropy=–(2/6)log2(2/6)–(4/6)log2(4/6)=SplittingBasedonInformationParentNode,pissplitintokpartitions;niisnumberofrecordsinpartitioniMeasuresReductioninEntropyachievedbecausethesplit.Choosethesplitthatachievesmostreduction(maximizesGAIN)UsedinID3andDisadvantage:Tendstoprefersplitsthatresultinnumberofpartitions,eachbeingsmallbutSplittingBasedonGainParentNode,pissplitintokpartitionsniisthenumberofrecordsinpartitioniAdjustsInformationGainbytheentropyofthepartitioning(SplitINFO).Higherentropypartitioning(largenumberofsmallpartitions)ispenalized!UsedinDesignedtoovercomethedisadvantageofInformationSplittingCriteriabasedonClassificationClassificationerroratanodetErrorError(t)=1-maxP(i|MeasuresmisclassificationerrormadebyaMaximum(1-1/nc)whenrecordsareequallydistributedamongallclasses,implyingleastinterestinginformationMinimum(0.0)whenallrecordsbelongtooneclass,implyingmostinterestinginformationExamplesforComputingErrorError(t)=1-maxP(i|06P(C1)=0/6= P(C2)=6/606Error=1–max(0,1)=1–1=15P(C1)= P(C2)15Error=1–max(1/6,5/6)=1–5/6=24P(C1)= P(C2)24Error=1–max(2/6,4/6)=1–4/6=Fora2-classMisclassificationErrorvs73Gini= NodeNodeNode=1–(3/3)2–==1–(4/7)2–=

34033403+7/10*=GiniimprovesTreeGreedySplittherecordsbasedonanattributetestthatoptimizescertaincriterion.DeterminehowtosplittheHowtospecifytheattributetestHowtodeterminethebestDeterminewhentostopStoppingCriteriaforTreeStopexpandinganodewhenalltherecordsbelongtothesameclassStopexpandinganodewhenalltherecordshavesimilarattributevaluesEarlytermination(tobediscussedDecisionTreeBasedInexpensivetoExtremelyfastatclassifyingunknownEasytointerpretforsmall-sizedAccuracyiscomparabletootherclassificationtechniquesformanysimpledatasetsExample:Simpledepth-firstUsesInformationSortsContinuousAttributesateachNeedsentiredatatofitinUnsuitableforLargeDatasets.–Needsout-of-coresorting.YoucandownloadthesoftwarePracticalIssuesofMissingCostsof500circularand500triangulardatapoints.Circular 0.5￡sqrt(x2+x2)￡ Triangularpoints:sqrt(x12+x22)>0.5orsqrt(x12+x22)<1Underfitting:whenmodelistoosimple,bothtrainingandtesterrorsareOverfittingduetoDecisionboundaryisdistortedbynoiseOverfittingduetoInsufficientLackofdatapointsinthelowerhalfofthediagrammakesitdifficulttopredictcorrectlytheclasslabelsofthatregion-InsufficientnumberoftrainingrecordsintheregioncausesdecisiontreetopredictthetestexamplesusingothertrainingrecordsthatareirrelevanttotheclassificationtaskNotesonOverfittingresultsindecisiontreesthataremorecomplexthannecessaryTrainingerrornolongerprovidesagoodestimateofhowwellthetreewillperformonpreviouslyunseenrecordsNeednewwaysforestimatingRe-substitution（重置換）errors:errorontrainingSe(tGeneralizationerrors:errorontesting(SMethodsforestimatinggeneralizationOptimisticapproach:e’(t)=PessimisticForeachleafnode:e’(t)=Totalerrors:e’(T)=e(T)+N·0.5(N:numberofleaf Foratreewith30leafnodesand10errorsontraining(outof1000instances):Trainingerror=10/1000=Generalizationerror=(10+30·0.5)/1000=ReducederrorpruningusesvalidationdatasettoestimateOccam’sGiventwomodelsofsimilargeneralizationerrors,oneshouldpreferthesimplermodeloverthemorecomplexmodel Forcomplexmodels,thereisagreaterchancethatitwasfittedaccidentallybyerrorsindata Therefore,oneshouldincludemodelcomplexitywhenevaluatingamodelXy10Xy1001……1Xy????……? CostisthenumberofbitsneededforSearchfortheleastcostlyCost(Model)usesnodeencoding(numberofplussplittingconditionexkNkk被錯分的樣本總 ea是置信水 p(k,N)=Ckxk(1-x)N-Ns2=Nx(1- N(Nx,Nx(1-N(Nx,kxN(Nx,k-

N￡k-￡￡￡ (N+

)x2-(2Ne+z

)x+Ne2￡

(N,e,a)

ae+/2+zaa

z

++4N1+aNHowtoAddressPre-Pruning(EarlyStoppingStopthealgorithmbeforeitbecomesafully-grownTypicalstoppingconditionsforaStopifallinstancesbelongtothesameStopifalltheattributevaluesaretheMorerestrictiveStopifnumberofinstancesislessthansomeuser-specifiedStopifclassdistributionofinstancesareindependentoftheavailablefeatures(e.g.,usingc2test)Stopifexpandingthecurrentnodedoesnotimproveimpuritymeasures(e.g.,Giniorinformationgain).HowtoAddressGrowdecisiontreetoitsTrimthenodesofthedecisiontreeinabottom-upfashionIfgeneralizationerrorimprovesaftertrimming,replacesub-treebyaleafnode.Classlabelofleafnodeisdeterminedfrommajorityclassofinstancesinthesub-treeCanuseMDLforpost-ExampleofPost-Class=Class=Error=TrainingError(Beforesplitting)=10/30PessimisticClass=Class=Error==(9+4·0.5)/30= Class=8ClassClass=8Class=4Class=3Class=4Class=4Class=1Class=5Class=1ExamplesofPost-Optimistic CaseC0:C1:C0:C1:Don’tC0:C1:C0:C1:PessimisticDon’tprunecase1,prunecaseReducederror

CaseC0:C0:C1:C0:C1:Missingvaluesaffectdecisiontreeconstructioninthreedifferentways:AffectshowimpuritymeasuresareAffectshowtodistributeinstancewithmissingvaluetochildnodesAffectshowatestinstancewithmissingvalueisclassified123456789?=-0.3log(0.3)-(0.7)log(0.7)===032410

Spliton=-(2/6)log(2/6)–(4/6)log(4/6)==0.3(0)+(0.9183)=Gain=0.9·–0.551)=Distribute?12345678900+32+4ProbabilitythatRefund=YesisProbabilitythatRefund=Nois0324Assignrecordtothe0324withweight=ClassifyNew

??331041121 ProbabilitythatMarital=Marriedis < >

ProbabilitythatMarital={Single,Divorced}is OtherDataSearchTreeDataNumberofinstancesgetssmallerasyoutraversedownthetreeNumberofinstancesattheleafnodescouldbetoosmalltomakeanystatisticallysignificantSearchFindinganoptimaldecisiontreeisNP-Thealgorithmpresentedsofarusesagreedy,top-down,recursivepartitioningstrategytoinduceareasonablesolutionOtherDecisiontreeprovidesexpressiverepresentationforlearningdiscrete-valuedfunctionButtheydonotgeneralizewelltocertaintypesofBooleanfunctionsClass=1ifthereisanevennumberofBooleanattributeswithtruthvalue=TrueClass=0ifthereisanoddnumberofBooleanattributeswithtruthvalue=TrueForaccuratemodeling,musthaveacompleteNotexpressiveenoughformodelingcontinuousvariables–Particularlywhentestconditioninvolvesonlyasingleattributeat-a-Decision1yy0

xBorderlinebetweentwoneighboringregionsofdifferentclassesisknownasdecisionboundaryxDecisionboundaryisparalleltoaxesbecausetestconditioninvolvesasingleattributeat-a-timeOblique(斜Decisionxx+y<Class=ClassTestconditionmayinvolvemultipleMoreexpressiveFindingoptimaltestconditioniscomputationallyTreeSamesubtreeappearsinmultipleModelMetricsforPerformanceHowtoevaluatetheperformanceofaMethodsforPerformanceHowtoobtainreliableMethodsforModelHowtocomparetherelativeperformanceamongcompetingmodels?ModelMetricsforPerformanceHowtoevaluatetheperformanceofaMethodsforPerformanceHowtoobtainreliableMethodsforModelHowtocomparetherelativeperformanceamongcompetingmodels?MetricsforPerformanceFocusonthepredictivecapabilityofamodel–Ratherthanhowfastittakestoclassifyorbuildmodels,scalability,PREDICTEDPREDICTEDabcda:TP(truepositive)b:FN(falsenegative)c:FP(falsepositive)d:TN(truenegative)MetricsforPerformancePREDICTEDMostwidely-used a+b+c+

TP+ LimitationofConsidera2-classNumberofClass0examples=NumberofClass1examples=Ifmodelpredictseverythingtobeclass0,accuracyis9990/10000=99.9%Accuracyismisleadingbecausemodeldoesnotdetectanyclass1exampleCostC(i|j):CostofmisclassifyingclassjexampleasclassComputingCostofPREDICTED+-+-10PREDICTED+-+-PREDICTED+-+-5Accuracy=Cost= Accuracy=Cost=CostvsPREDICTEDabPREDICTEDabcdN=a+b+c+dAccuracy=(a+PREDICTEDpqqpCost=p(aPREDICTEDpqqp=p(a+d)+q(N–a–=qN–(q–p)(a+=N[q–(q-p)·

=

=F-measure(F)=2rp r+ PrecisionisbiasedtowardsC(Yes|Yes)&RecallisbiasedtowardsC(Yes|Yes)&F-measureisbiasedtowardsallexcept wa+ wa+wb+w3c+ wModelMetricsforPerformanceHowtoevaluatetheperformanceofaMethodsforPerformanceHowtoobtainreliableMethodsforModelHowtocomparetherelativeperformanceamongcompetingmodels?MethodsforPerformanceHowtoobtainareliableestimateofPerformanceofamodelmaydependonotherfactorsbesidesthelearningalgorithm:ClassCostofSizeoftrainingandtestLearningcurveshowshowaccuracychangeswithvaryingsamplesizeRequiresasamplingscheduleforcreatinglearningcurve:(Langley,etal)Geometricsampling(Provostetal)EffectofsmallsampleBiasintheMethodsofReserve2/3fortrainingand1/3forRandomRepeatedCrossPartitiondataintokdisjointk-fold:trainonk-1partitions,testontheremaining StratifiedoversamplingvsSamplingwithModelMetricsforPerformanceHowtoevaluatetheperformanceofaMethodsforPerformanceHowtoobtainreliableMethodsforModelHowtocomparetherelativeperformanceamongcompetingmodels?ROC(ReceiverOperatingDevelopedin1950sforsignaldetectiontheorytoanalyzenoisysignalsCharacterizethetrade-offbetweenpositivehitsandfalsealarmsROCcurveplotsTP(onthey-axis)againstFP(onthex-axis)PerformanceofeachclassifierrepresentedasapointontheROCcurvechangingthethresholdofalgorithm,sampledistributionorcostmatrixchangesthelocationofthepointROC1-dimensionaldatasetcontaining2classes(positiveandanypointslocatedatx>tisclassifiedasAtthresholdTP=0.5,FN=0.5,FP=0.12,ROC(0,0):declaretobenegative(1,1):declareeverythingtobepositiveclass(1,0):DiagonalRandomBelowdiagonalpredictionisoppositeofthetrueclassUsingROCforModelNomodelconsistentlyoutperformtheotherM1isbetterforsmallFPRM2isbetterforlargeFPRAreaUndertheROCArea=RandomArea=HowtoConstructanROC1+1+2+3-4-5-6+7-8+9-+SorttheinstancesaccordingtoP(+|A)indecreasingorderApplythresholdateachuniquevalueofP(+|A) TN,FNateachthresholdTPrate,TPR=FPrate,FPR=FP/(FP+HowtoconstructanROC++-+---+-++ld544333322105544321100000112344555011222233451011000ROCTestofGiventwoModelM1:accuracy=85%,testedon30ModelM2:accuracy=75%,testedon5000CanwesayM1isbetterthanHowmuchconfidencecanweplaceonaccuracyofM1andM2?Canthedi