人體三維姿勢的恢復(fù)與重建

YingYang:Restorationandreconstructionofthree-dimensionalhumanbodypostureChapteroneintroduction1.1ResearchpurposeandsignificanceInthefieldofcomputervision,therecoveryandreconstructionofhumanpostureistorecoverhumanposturecharacteristicsfromasingleperspectiveormultipleperspectives(unsynchronized,synchronized)imagesorvideo.Humanposturereconstructionisamotion-sensingtechnology,ithasspecialsciencebackground,importantacademicsignificanceandwideapplicationprospect,sohasalwaysbeenoneofthemostactiveresearchdirectioninthefieldofcomputervision,thisdirectionhasbeenattractingtheattentionofscholarsandindustrysupport,itisstillacurrentoneofthemostactiveareaofresearch.Bodyposturereconstructionmainlyincludesbodyrecognitionreconstruction,gesturerecognitionreconstructionandfacialposturerecognitionreconstruction[1].Itsapplicationcanbedividedintothreeaspects:monitoring,analysisandcontrol[1].Monitoringistobeabletoautomaticallymonitorandreasonaboutpossibleevents.Theapplicationofintelligentmonitoringcanimprovethequalityofmonitoringbyidentifyingpeopleandidentifyingfingerprints.Analysisisgenerallyusedtoidentifyandidentifyindividualsorgroupsofbehavioralpurposes;Controlisgenerallyusedformotionestimationofhumanbody.Human-computerinteractionisoneofthemostimportantapplicationsofbodyposturereconstructiontechnology。Becauseoftheimportantresearchvalueofbodyposturereconstruction,bodypostureestimationhasbeenwidelyusedinmanyaspectsandhasmadegreatachievements.Inthegameanimation,filmandtelevisionspecialeffects,human-computerinteraction,virtualrealityandsoon,thebirthofthebodyfeelinggame,andthroughthebodyposturetocapturethemovementsoftheactors,mapittothecorrespondinghumanbodymodel,afterthesubsequentprocessinggetvividcartoonimageor3deffects,soastosatisfythedemandsofpeopletoworklifelike.Intermsofsportsanalysis,oneoftheapplicationsofposturereconstructionistorecordthemovementstatusofvariousbodypartsofusersinrealtime,generateaccuratedatareports,andprovidereal-timesportsstatusanalysisandSuggestionsthroughtheAPP,soastoenableuserstoachieveabettereffectofphysicalexercise.Intermsofmedicaldiagnosis,Abdolrahim[2]realizedthetrackingandmodelingofpatients'bonesintheoperationprocessintheoperatingroomenvironmentthroughKinectcamera,enablingdoctorstoobservepatients'conditionsandtheoperationprocessinrealtime,thusimprovingthesafetyoflarge-scalesurgery.1.2ResearchprogressanddevelopmentstatusathomeandabroadResearchonbodyposturerestorationandreconstructionbeganinthe1980s,andgreatprogresshasbeenmadeinthepastfewdecades.Thebodyposturereconstructioncanbedividedintotwo-dimensionalbodypostureestimationreconstructionandthree-dimensionalbodypostureestimationreconstruction.Accordingtothedifferentnumberofthesinglebodyposturereconstruction,itcanbedividedintosingle-personbodyposturereconstructionandmulti-personbodyposturereconstruction.Atlast,accordingtothetimeneededforhumanbodyposturereconstruction,itcanbedividedintoreal-timehumanbodyposturereconstructionandnon-real-timehumanbodyposturereconstruction.Intherestorationandreconstructionoftwo-dimensionalmultiplayerhumanbodyposture,therearetop-downandbottom-upmethods.Top-downmethod[3]-[7]useshumanbodydetectortodetectallpeopleintheimageandindependentlyperformsingle-personposturereconstructionforeachdetection.Thesetop-downmethodsdirectlymakeuseoftheexistingsingle-personhumanbodyposturereconstructiontechnology[8]-[14],buttheaccuracyofhumanbodydetectorisneededasaguarantee,thatis,ifpeoplegetclosetoeachother,thehumanbodydetectorwillfailtodetectandcannotrecover,thenthesingle-personhumanbodyposturereconstructioncannotbeperformed.Inaddition,therunningtimeofthetop-downapproachisproportionaltothenumberofpeopleinthedatastream:foreachtestrunofasinglehumanposturereformer,themorepeople,thehigherthecomputationalcost.SuchasGeorge[7]touseFaster--RCNN(adetectionalgorithm)asthedetector,detectmorethanonepersonintheimage,andthedetectingboxdocuttogeneratethesamesizeoftheinputimage,thenUSEStheFullyConvolutionalResNet(completelyconvolutionresidualnetwork)forthecharactersinatestboxtopredictDenseHeatmap(intensiveheatmaps)andOffset(deviation),throughtheDenseHeatmapandOffsetfusionpointofaccuratepositioning.Theworkofhao-shuFang[15]ismainlytooptimizetheimperfectPoseEstimationofthehumanbodydetectedinthetop-downmethod.ThemethodistoaddParallelSinglePersonPoseEstimation(ParallelSPPE)tofurtherstrengthenSpatialTransformerNetworks(STN)soastoextractthehighqualityposes.Incontrast,thebottom-upapproachisunaffectedbythehumandetector,providingsomerobustnessandpotentiallymakingtheruntimecomplexityindependentofthenumberofpeopleinthedatastream.However,thepreviousbottom-upapproach[16]-[17]cannotdirectlyutilizetheglobalcontextinformation,becausethefinalidentificationprocessrequiresexpensiveglobalinferencecostandcannotguaranteetheefficiencygaininpractice.Forexample,Pishchulin[16]etal.proposedabottom-upmethod.First,thedeepneuralnetworkwasusedtofindoutallcandidatehumanbodypartspoints,andthesepointswereformedintoamap.Atthesametime,eachnodeismarked,whichpartofthebodyisclassified,andtheresultisoutputbycombiningthetwo.However,thismethodisequivalenttosolvingtheintegerlinearprogrammingproblemonacompletelyconnectedgraph.InadditiontoDeepCut[16],Insafutdinov[17]etal.addedmoreaccuratepartialdetectorbasedonResNet[18]andrelevantimagematchingscoretechnology,greatlyreducingtherunningtime.However,therecognitionofeachimageinthismethodstillneedsseveralminutes,andthenumberofpeoplewhocanberecognizedislimited.Therepresentationofthepairedscoresusedinthispaperisdifficulttoaccuratelyregression,andaseparatelogisticregressionneedstobeadded.Inthemainstreamdomain,3dposeestimationisthemain3djointproblemofbodypositioningfromimage,videosequence,singleviewormultipleview.Three-dimensionalattitudeestimationmethodscanbedividedintotwocategories:two-stagemethodanddirectestimationmethod.Thetwo-stagemethodfirstlyUSESthetwo-dimensionalposturedetector[19]-[21]ortheactualtwo-dimensionalpostureonthegroundtolocatethetwo-dimensionaljointposition,andthenpredictsthethree-dimensionaljointpositionfromthetwo-dimensionaljointbyregression[26][28]orsimulationfitting.Zhouetal[21][23]usedthetwo-dimensionalthermalmapfromtheConvNet(convolutionalnetwork)toreconstructthethree-dimensionalposeinthevideosequence.Bogoetal[24]fittedthestatisticalmodelof3dhumanbodyshapetothepredicted2djoint.Therearealsosomenetworksthatcanimprove2dattitudeestimationto3dattitudethroughprocessing[25]-[27].ItisworthnotingthatMartinezetal[28]designedasimplemultilayerperceptron,whichrestoresthe3djointpositiongivenatwo-dimensionalkeypointasinput.Throughthismultilayerperceptron,theexistingbestresult(state-of-the-art)isobtained.Thedirectestimationmethodistoinputapictureanddirectlyestimatethecoordinatesof3dnodeswithoutgoingthroughtheprocessoffirstestimating2dnodes.Thisismainlyduetothevideodatasetwithlivegroundmotioncapture,suchasHumanEva[22]andhuman3.6m[29].Thetrainingdataprovidedbythedatasetcanmakethe3djointestimationproblemastandardsupervisedlearningproblem.So.Manyrecentmethodsdirectlyestimate3dnodecoordinatesinthedeeplearningframework[30]-[34].Atpresent,themainstreammethodisstillcompleteconvolution,whiletherecentlyreturned3djointcoordinatesofXiaoetal[35]basicallyachieveexcellentresultsin3dposture.Themainproblemwiththesedirectestimationmethodsistocapture3dannotatedimageswithaccurategroundrealityinacontrolledMoCap(laboratory)environment.Themodeltrainedonlyontheseimagescannotbewellappliedtootheroutdoorscenes.1.3ThemaincontributionandinnovationofthispaperNow,though,withthesuccessofconvolutionalneuralnetworks,thereisanend-to-endsystemfor3dattitudeestimation,whichcanpredict3djointpositionsgiventheoriginalimagepixels.Althoughsuchasystemhasexcellentperformanceandenablesreal-timeinteraction,whenthepredictionresultsarenotaccurate,itisdifficulttodeterminewhethertheerrorresultsfromlimitedtwo-dimensionalposture(visual)understandingorfrommappingtwo-dimensionalposturetothree-dimensionalposture.Therefore,thesystemstilladoptstheideaoftwo-stagemethod.Firstly,thetwo-dimensionalcoordinateisestimatedbythetwo-dimensionalattitudeestimationalgorithmbasedonpartialaffinityfields,andthenasimplemultilayerperceptronnetworkdesignedbyMartinezetal[28]isusedtomakethethree-dimensionaljointpositionbereturnedunderthegiventwo-dimensionalkeypointsasinput.IntheworkofMartinezetal[28].acompletetwo-stageapproachisadopted.Afterinputvideo,theestimationof2dkeypointsofallframesiscompleted,andtheninputthenetworktoestimate3dkeypointsfrom2dkeypoints.Thissystemimprovesthispoint.Intermsofthefunctionalityofthesystem,3djointcoordinatescanbedirectlyoutputinthecaseofgivenoriginalimagepixels,buttheprocessfrominputto2djointto3djointhasgonethroughinthemiddle.Afterfinishingthethree-dimensionaljointpointestimatesinthispaper,andthenthethreedimensionaljointcoordinatesusedinunity3dcharactermodelofdriver,forunity3dengineusingexternalprogramfordatatransmissionhasavarietyofways,afterasaplug-in,externalexecutablecallsandtheeffectofnetworkprogramming,programmingornetworkdatatransmissioneffectisbest,andinthiswaytoachievethebodyfeelingtechnologybasedoncomputervision.Therefore,themaininnovationsandcontributionsofthispaperareasfollows:1、Martinezetal[28]adoptedacompletelytwo-stagemethodtocompletethe2-dattitudeestimationofallframes,andtheninputthenetworktorestorethethree-dimensionalkeypointcoordinates.Asaresult,thesystemthatcanonlyconductbatchprocessingtoestimatethree-dimensionalhumanjointcoordinateshasreal-timeinteractionfunction.2、Theapplicationof3dhumanpostureestimationinmotionsensingtechnologywasrealized,andasystemwithgoodeffectandreal-timeinteractionabilitywasdesignedtodriveunity3dcharactermodelmotionbyinputtingvideocharactergesture.3、Itprovidesanewideafortheapplicationof3dhumanpostureestimation.1.4PaperstructurearrangementInthispaper,arestorationandreconstructionalgorithmbasedon3dhumanpostureisdesignedandimplementedtodrivetheunity3dcharactermodelmovementsystem.Basedonthereconstructionandrestorationalgorithmofhumanposture,thissystemestimatesthe3dhumancoordinateofthefigureinthepicturethroughasinglepicture,andtransmitsthiscoordinatetounity3dtodrivetheunity3dcharactermovement.Specifically,thispaperisdividedintofivechapters.Thefirstchapteristheintroductionpart,firstanalyzestheresearchpurposeandsignificanceofthesubject;Secondly,theresearchprogressanddevelopmentstatusathomeandabroadareintroduced.Thensummarizetheinnovationandcontributionofthispaper;Finally,theresearchcontentandorganizationalstructureofthispaperareintroduced.Thesecondchaptermainlyintroducessomeknowledgeusedinthispaper.Firstly,itintroducesCOCOandhuman3.6m,twoimportantdatabasesinthefieldofhumanpostureestimation,aswellasthedivisionofvariouspartsofhumanbody.Atthesametime,itintroducestheskeletonofunity3dcharactermodel.ThenthenetworkstructureVGGNetwhichisusedtoextractthefeatureswhenestimatingtherecoveryoftwo-dimensionalcoordinatesisintroducedindetail.Finallyintroducesthecorequaternionandrotationthatmakethecharactermodelmove.Thethirdchapterisadetailedintroducesthesystemhowtorealizetheparts,thefirstdescribestheoverallframeworkofthewholesystem,thenintroducestherecoveringfromRGBimagealgorithmtoestimatethetwo-dimensionaljointcoordinates,thealgorithmisbasedonpartialaffinityfieldoftwodimensionalhumanbodypostureestimation,afterintroducedtheprocessofthisalgorithm,thisalgorithmisactuallyobserveabottom-upthought,sofirstusingconvolutionneuralnetworkconcludesthatallthepeopleallofthekeypoints,andthenbyHungaryandKMalgorithmjointpointtoconnecttothesameperson,thancompleteparsing.Therefore,thispaperUSESnetworkarchitecture,featuregraphextraction,confidencegraphdetection,partialaffinityfield,andpartialaffinityfieldformulti-personanalysistodescribetheimplementationprocessindetail.Then,thealgorithmofrecoveringandestimating3djointcoordinatesfrom2dkeypointsisintroducedindetail.Thispartisactuallyverysimple,thatis,aregressionnetworkisused.Therefore,wefirstintroducethenetworkarchitecture,andthenexplaintheimportantdataprocessing.Finally,thealgorithmtodrivethetargetactionisdescribed.Thefourthchapteristhefulltextoftheexperiment,thefirstistherestorationofthree-dimensionalcoordinatesoftheexperiment,thispartoftheexperimentrespectivelyconductedqualitativeandquantitativeexperiments,quantitativeexperimentsonthehuman3.6mdataset,comparativeanalysisofthispaperandpreviouswork.Sinceourmodelofrestoring3dcoordinateswastrainedonhuman3.6m,whichwasmeasuredinanindoorenvironment,qualitativeexperimentsonMPII,amulti-personoutdoordataset,summarizedthelimitationsofthemethodofrestoring3djointcoordinatesinthispaper.Forthepartofdrivingunity3dcharactermodel,thereisnoindextomeasurethequalityoftheresults,sothispaperonlyconductsaqualitativeexperiment,andsummarizesthelimitationsofthemethodinthisstep.Finally,thispaperdoesacompleteprocesstesttoverifythefeasibilityofthisworkinpracticalapplication.Chapterfiveisthesummaryandprospectofthewholepaper.Firstly,theworkissummarized,andthenthelimitationsofthecurrentsystemandtheprospectofthefutureworkareanalyzed.Chaptertwointroducesrelevanttheories2.1Bodypartsdivisionandunityfigureskeleton2.1.1COCOdatasetanditshumanbodynodeMSCOCOisatwo-dimensionaldataset,whichcanbeusedforimagerecognition,ObjectDetectionandimageSegmentation.ThedatawascollectedusingAmazonMechanicalTurkFlickronFlickr,andtheimagesincluded80objectcategoriesandvariousscenarios.COCOdatasetsarealsodividedintotrainingsets,validationsets,andtestsets.Asshowninfigure2.1,thekeypointsandindexesofhumanbodyduringtheestimationoftwo-dimensionalpostureinthispaperareshowninfigure2.1.Thereare18humankeypointsand19partialaffinityfieldsFigure2.1correspondingnodesoftwo-dimensionalhumanbodyThenumbersandcorrespondingkeypointsofhumanbodyareshownintable2.1,andthenumbersandtheircorrespondingaffinityfieldsareshownintable2.2:Table2.1specificbodypartscorrespondingtoNumbersSerialnumberPartsofthebodySerialnumberPartsofthebody0Nose9RKnee1Neck10RAnkle2RShoulder11LHip3RElbow12LKnee4RWrist13LAnkle5LShoulder14REye6LElbow15LEye7Lwrist16REar8RHip17LEarTable2.2numberingandcorrespondingpartialaffinityfieldsSerialnumberCorrespondingpartaffinityfieldsSerialnumberCorrespondingpartaffinityfields1NeckRShoulder11LHipLKnee2NeckLShoulder12LKneeLAnkle3RShoulderRElbow13NeckNose4RElbowRWrist14NoseReye5LShoulderLElbow15REyeREar6LElbowLwrist16NoseLeye7NeckRHip17LEyeLear8RHipRKnee18RShoulderREar9RKneeRankle19LShoulderLEar10NeckLHip2.1.2Thehuman3.6mdatasetanditshumanbodynodesHuman3.6mprovides3.6mimagestakeninalaboratorysetting.Theimagesaretakenbyfourcamerasfromfourdifferentanglesofthesamesceneandcontainthree-dimensionalkeyinformation.Atotalof11actorsinthedataset,includingfiveactressesandsixmales,performed15differentactionsincluding:pointing,discussing,eating,greeting,calling,posing,buying,sitting,sitting,smoking,takingphotos,waiting,walking,walkingthedog,andwalkingingroups.Inmanypapers,dataofS1,S5,S6,S7andS8areusedastrainingdata,anddataofS9andS11areusedastestdata.CollectionSettingsareshowninfigure2.2.Figure2.2picturecollectionSettings.Theareaof3mand4misthedataacquisitionarea.Thereisacalibratedcolorcameraateachcorneroftheacquisitionarea.Thereare4motioncapturecamerasonthewallsonbothsides,and2onthehorizontalareabelow.Human3.6mactuallyhasatotalof32keypoints,butinfactweuseonly14or17nodesfortrainingortesting.Inthisarticle,weuse17nodes,whichhavetheordinalnumberandcorrespondingjointsofhuman3.6masshownintable2.3below.Table2.3human3.6mdatacorrespondingtojointsnumberCorrespondingtothejointsnumberCorrespondingtothejoints0Hip14Neck/Nose1RHip15Head2RKnee17LShoulder3RFoot18LElbow6LHip19LWrist7LKnee25RShoulder8LFoot26RElbow12Spine27RWrist13Thorax2.1.3Characterskeletoninunity3dInunity3d,thereisananimationmodelcalledHumanoid,forwhichunity3dalsoprovidesanextendedHumanoidanimationtoolset.Duetothesimilarityofbonestructure,animationcanbemappedfromonehumanoidskeletontoanothertoachieveredirectionandinversekinematics.Humanoidmodelsshouldhavethesamebasicstructure,withthemainjointsofthebody,headandlimbs.Therefore,whentheanimatedcharactermodelisimportedintounity3d,theskeletonmappingwithHumanoidtypeanimationneedstobecompletedfirst.Asshowninfigure2.3,thefollowingfigureisthebonemappingoftheHumanoidtypeofunity3d.Figure2.3skeletonmappingofHumanoidtypesWhenthemodelcharacterofunity3dtargetFBX(filedatastoredintreestructure)doesnotbindbones,3danimationproductionsoftware(blender,3dmax,etc.)canbeusedtobindbones.Figure2.4showsthemodelfigureboundtotheskeleton.Theskeletonisshownafteropeninginblender.Figure2.4blenderbindingfiguremodelskeletondisplay2.2VGGNetnetworkdetailsVGGNet[36]isakindofconvolutionalneuralnetwork,whichreducedthetop-5errorrateby7.3%intheILSVRC2014competitionandachievedthesecondplace.ThenetworkhasbeendevelopedbyOxfordUniversity'sVisualGeometryGroupandGoogleDeepMind.VGGNetdeepensthenetworkstructuretoimproveperformancebyrepeatedlystacking3×3convolutionkerneland2×2maximumpoolinglayer[37].VGGNethas5convolutions,andeachconvolution(including2to4convolutionlayers)willbeconnectedwithamaximumpoolinglayertoreducethesizeoffeaturemapaftertheend[49].Figure2.6showsthenetworkconfigurationofVGGNetatalllevels,andtable2.4showsthenumberofparametersofVGGNetatalllevels.Inordertoreducethenumberofparametersandhavemorenonlineartransformswiththesamesizeofreceptivity,theconvolutionkernelineachsegmenthasthesamesize,andthemoreconvolutionkernelinthelowersegment[48].Asshowninfigure2.6,theseriesconnectionoftwo3×3convolutionlayersisequivalenttoa5×5convolutionlayer,thatis,thesizeofthereceptivefieldbecomes5×5[48].Thesuperpositionofthree3×3convolutionlayersisequivalenttoa7×7convolutionlayer.Thenumberofparametersoftheformerisonlyhalfthatofthelatter,andtheformercanuseReLUactivationfunctionthreetimes,whilethelattercanonlyuseitonce,whichenhancesthelearningabilityoftheconvolutionalneuralnetworkforfeatures[37].Figure2.5two3×3convolutionlayersinseriesareequivalenttoa5×5convolutionlayerFigure2.6VGGNetnetworkconfigurationatalllevelsTable2.4numberofparametersofVGGNetnetworks(millions)ConvolutionalneuralnetworkA,A-LRNBCDEThenumberofarguments1331331341381442.3QuaternionsandrotationsInunity3d,boththepersonmodelandtheobjectareTranformcomponents.Thiscomponentcontrolstherotationvariablecalledrotationanditstypeisquaternion.Inthissystem,thecoreofunity3dcharactermovementistocalculatetherotationquaternionofeachjointboneofthecharacterrelativetoitsparentbonenode.Theskeletonrelationshipofunity3dmodelhasbeenexpoundedintheprevioussection.Thissectionfocusesonwhatquaternionsareandhowtheyrotate.2.3.1DefinitionsandpropertiesofquaternionsQuaternionsaredividedintorealandimaginaryparts.Forexample,informula2.1,istherealpartandistheimaginarypart.Allquaternions(HisforWilliamRowanHamilton,thediscovererofquaternions)canbeexpressedasfollows:公式以章為單位編號（2.1）公式以章為單位編號Amongthem:（2.2）ExpressedasacombinationofrealNumbersandvectors:（2.3）Theoperationalpropertiesofquaternionsare:Quaternionmultiplication:(2.4)Conjugatequaternion:（2.5）Squaremodulusofquaternion:（2.6）Inverseofaquaternion:Theinverseofquaternionmultiplicationisdefinedasor.andistheInverseofquaternionsWeprovidedthat:where,theInverseofquaternionscanbeobtainedbycalculatingandreasoning:（2.7）2.3.3QuaternionsrepresentrotationsRigidbodyrotationinthree-dimensionalspacecanbedescribedasfollows:Anypointontherigidbodyrotatesaroundtheaxisthatpassesthroughtheorigin.FindtherotatedpointQuaternioncanbeusedtodescriberotationinthree-dimensionalspace,Themathematicaldescriptionofquaternionqisasfollows:（2.8）Thisquaternionrepresentstherotationofthetaaroundtheaxisrepresentedbytheunitvector(x,y,z),soapointinrigidbodycoordinatesintheformofaquaternionis,therotationofthetaaroundtheaxis(x,y,z)is:（2.9）Thecompositionrotationcanbeobtainedbyleftmultiplicationofnewquaternionssimilartothecompositionmatrix.Chapterthreerealizationofeachpartofthesystem3.1OverallflowofthesystemFramediagram3.1showstheframeworkofoursystem.Thewholesystemcanbedividedintoserver-sideandclient-side.Theserver-sidefunctionistoinputpicturesandfinallyget3dkeypoints.Thefunctionoftheclientistoacceptthecoordinatesof3dkeypoints,andthencalculatetherotationquaternionofeachjointrelativetoitsparentnodeaccordingtothecoordinatestotransformintothecorrespondingposture.DatafromtheserveristransmittedtotheclientthroughUDPprotocol.Figure3.1TheoverallframeworkofthesystemFlowchart3.2showstheprocessingprocessofasingleframeimage.Thissystemprocessesvideo.Inordertooutputthecontinuityandcoherenceofthree-dimensionalposeaction,adjacentframesaresmooched.Theinputvideoisprocessedbyopencv(opensourcedistributedcross-platformcomputervisionlibrary)andconvertedintokeyframes.Thefollowingprocessisshowninfigure3.2.Thetotalisdividedintotwoparts,theserversideandtheclientside.Serverisdeployedinc++3dhumanbodyposturerecoveryestimationprocess,gettheestimatedthree-dimensionaljointpointdata,usingUDP(userdatagramprotocolfordatatransmissiontotheclient,unity3denginefortheclient,receivesthethreedimensionaljointpointcoordinates(targetattitude),accordingtothetargetmovementalgorithmtocalculatethedrivingcharactersdojointpointrelativetotheirparentnoderotationquaternion,makethetargetaction.Ontheserverside,theinputimageiscalculatedbythetwo-dimensionalattitudeestimationalgorithm,andthethree-dimensionalnodecoordinatesareestimatedbyasimpleregressionnetwork.Thefollowingsectionswillfocusonthetwo-dimensionalnodealgorithm(section3.2ofthisarticle),thethree-dimensionalnodealgorithm(section3.3ofthisarticle)andtheunity3dalgorithm(section3.4ofthisarticle)thatdrivesthetargetactionofthecharacter.Figure3.2systemprocessdetaildiagram3.2Two-dimensionalhumanposturereconstructionbasedonpartaffinityfieldTwo-dimensionalhumanposturereconstructionbasedonpartialaffinityfields[44]isabottom-uptwo-dimensionalpostureestimationalgorithm,whichisoneofthealgorithmsusedtoidentifymultiplehumanpostureinrealtime.Top-downalgorithmsaremainlycomposedofhumandetectorandsingleattitudeestimation.Thesemethodsrelyonthesetwopartsforperformance.Ifhumandetectorfailstodetect(whenmultiplepeopleareclosetoeachother),thesemethodshavegreatlimitations.However,whentherearemorecharacterstobecalculated,thecalculationwillbeslowandreal-timeperformanceisnotgood.Thetwo-dimensionalhumanposturereconstructionalgorithmbasedonpartialaffinityfieldaddspartialaffinityfield,whichisagroupoftwo-dimensionalvectorfieldsencodingthepositionanddirectionoflimbsintheimage.Thesefields,alongwiththeconfidencemapofthejoint,arecombinedbyCNNforlearningandprediction.3.2.12djointpredictionoverallflowFigure3.3two-dimensionalnodepredictionflowchartFigure3.3isaschematicdiagramofthewholeprocessoftwo-dimensionalhumanposturereconstructionbasedonpartialaffinityfields.Theentiremodelisbuiltfromthebottomup,identifyingkeypointsandjointdomains,andthencombiningthemintoindividualposturediagramsthroughalgorithms.ThewholemodelincludestwoCNNnetworksintotal.ThegoalofthesetwoCNNnetworksistofindtheaffinityfields(associationareas)ofalltherelationnodesandtherelationnodes,andthenconductvectorconnectionaccordingtothekeypointsandassociationareas.(1)Findal