亞馬遜IBM 《高效神經(jīng)架構(gòu)搜索》教程 -Efficient Neural Architecture Search

EfficientNeuralArchitectureSearchpatiAmazonWebServices,IBMResearchyEfficientEfficientNeuralArchitectureSearchMartinMartinWistuba,TejaswiniPedapati,AmazonWebServices,IBMResearch25July20221/104EfficientEfficientNeuralArchitectureSearchIntroduction1.Introduction3.Zero-ShotNASLearningforNASMartinMartinWistuba,TejaswiniPedapati,AmazonWebServices,IBMResearch25July20222/104EfficientEfficientNeuralArchitectureSearchIntroductionsImagestakenfromSimoneBiancoetal.“BenchmarkAnalysisofRepresentativeDeepNeuralNetworkArchitectures”.In:IEEEAccess6(2018),pp.64270–64277,ChristianSzegedyetal.“Inception-v4,Inception-ResNetandtheImpactofResidualConnectionsonLearning”.In:ProceedingsoftheThirty-FirstAAAIConferenceonArtificialIntelligence,February4-9,2017,SanFrancisco,California,USA..2017,pp.4278–4284MartinMartinWistuba,TejaswiniPedapati,AmazonWebServices,IBMResearch25July20222/104EfficientEfficientNeuralArchitectureSearchIntroductionMartinMartinWistuba,TejaswiniPedapati,AmazonWebServices,IBMResearch25July20223/104EfficientEfficientNeuralArchitectureSearchIntroductionPart1ce?One-shottechniquesinNAS?Overview?Shortcomings?OtherflavoursPart2?EffectiveNASwithtransferlearningapproachesbasedon?TransferNASoptimizers?Few-ShotNASoptimizersrningCurveRankingMartinMartinWistuba,TejaswiniPedapati,AmazonWebServices,IBMResearch25July20224/104d-datasetEfficientNeuralArchitectureSearchIntroductionrningProblem(1)Λ(α,d)=argminL(mα,θ,dtrain)+R(θ)(1)mα,θ∈Mα?α-neuralarchitectureNASProblemα*=argmaxO(Λ(α,dtrain),dvalid)=argmaxf(α).(2)α∈Aα∈AAsearchspaceMartinMartinWistuba,TejaswiniPedapati,AmazonWebServices,IBMResearch25July20225/104EfficientEfficientNeuralArchitectureSearchIntroductionral?Thelimitationtoasubspaceallowsforconsidering?humanexpertknowledge,?specifictask(e.g.mobilearchitectures)and?Wedistinguishtwotypesofsearchspaces:?globalsearchspaceMartinMartinWistuba,TejaswiniPedapati,AmazonWebServices,IBMResearch25July20226/104 xzEfficientNeuralArchitecture xzEfficientNeuralArchitectureSearchIntroductionthesametopology x×n×n×nz(n)z(n)z(n?2)xx(1)z(n?1)z(n)z(n+1)x(1)z(n?1)MartinMartinWistuba,TejaswiniPedapati,AmazonWebServices,IBMResearch25July20227/104Block1+EfficientNeuralArchitectureSearchBlock1+EfficientNeuralArchitectureSearchIntroductionStructureofacell.Block3Block4+++Block2Block5Celli+Block3Block4+++Block2Block5Celli+ i?1i?2MartinMartinWistuba,TejaswiniPedapati,AmazonWebServices,IBMResearch25July20228/104EfficientEfficientNeuralArchitectureSearchIntroduction normalcell×n normalcell×n normalcell×nx(a)CIFAR-10 oftmax normalcell×n reductioncell normalcell×n reductioncell normalcell×n reductioncell×2 x(b)ImageNetMartinMartinWistuba,TejaswiniPedapati,AmazonWebServices,IBMResearch25July20229/104EfficientEfficientNeuralArchitectureSearchOne-ShotArchitectureSearch1.IntroductionMemoryConsumptionlapseDARTSDiscretizationncy2.3OtherflavoursofOne-shotNAS3.Zero-ShotNASLearningforNAS 25July202210/104EfficientEfficientNeuralArchitectureSearchOne-ShotArchitectureSearchOverview1.IntroductionMemoryConsumptionlapseDARTSDiscretizationncy2.3OtherflavoursofOne-shotNAS3.Zero-ShotNASLearningforNAS 25July202210/104EfficientEfficientNeuralArchitectureSearchOne-ShotArchitectureSearchOverviewWedistinguishseveralmethodsthatmaximizetheresponsefunction:?Reinforcementlearning:learntosampleαthatmaximizef.?Evolutionaryalgorithms:evolveαthatmaximizef.?Surrogatemodel-basedoptimization:approximatefbyf?anduseittomaximizef.?One-shotarchitecturesearch:learnonemodelanduseittomaxf.MartinMartinWistuba,TejaswiniPedapati,AmazonWebServices,IBMResearch25July202210/104OverviewEfficientNeuralArchitectureSearchOne-ShotArchitectureSearchOverviewUntilnow,andidatearchitectureistrainedfromscratchtoobtainvalidationNetMartinMartinWistuba,TejaswiniPedapati,AmazonWebServices,IBMResearch25July202211/104Overview21111234221ooooooooooooooEfficientNeuralArchitectureSearchOne-ShotArchitectureSearchOverview21111234221oooooooooooooo11221122221122MartinMartinWistuba,TejaswiniPedapati,AmazonWebServices,IBMResearch25July202212/104Overview21111234221ooooooooooooooEfficientNeuralArchitectureSearchOne-ShotArchitectureSearchOverview21111234221oooooooooooooo11221122221122MartinMartinWistuba,TejaswiniPedapati,AmazonWebServices,IBMResearch25July202212/104Overview211122121234221ooooooooooooooooooEfficientNeuralArchitectureSearchOne-ShotArchitectureSearchOverview211122121234221oooooooooooooooooo11221122221122MartinMartinWistuba,TejaswiniPedapati,AmazonWebServices,IBMResearch25July202212/104Overview21112212123422ooooooooooooooooooEfficientNeuralArchitectureSearchOne-ShotArchitectureSearchOverview21112212123422oooooooooooooooooo1122111122221122MartinMartinWistuba,TejaswiniPedapati,AmazonWebServices,IBMResearch25July202212/104Overview21111212123422ooooooooooooooooooEfficientNeuralArchitectureSearchOne-ShotArchitectureSearchOverview21111212123422oooooooooooooooooo1122111122221122MartinMartinWistuba,TejaswiniPedapati,AmazonWebServices,IBMResearch25July202212/104OverviewEfficientNeuralArchitectureSearchOne-ShotArchitectureSearchOverviewssentropylossofiscomputedonaminibatchoftrainingdataMartinMartinWistuba,TejaswiniPedapati,AmazonWebServices,IBMResearch25July202213/104OverviewEfficientNeuralArchitectureSearchOne-ShotArchitectureSearchOverviewzationLuoetalMartinMartinWistuba,TejaswiniPedapati,AmazonWebServices,IBMResearch25July202214/104EfficientEfficientNeuralArchitectureSearchOne-ShotArchitectureSearchOvervieworkHieuPhametal.“EfficientNeuralArchitectureSearchviaParameterSharing”.In:Proceedingsofthe35thInternationalConferenceonMachineLearning,ICML2018,Stockholmsm¨assan,Stockholm,Sweden,July10-15,2018.2018,pp.4092–4101MartinMartinWistuba,TejaswiniPedapati,AmazonWebServices,IBMResearch25July202215/104EfficientEfficientNeuralArchitectureSearchOne-ShotArchitectureSearchOverviewAlgorithm1ENASrsveryiterationdoelComputecross-entropyloss▽θEαonmforamini-batchoftrainingdataMartinMartinWistuba,TejaswiniPedapati,AmazonWebServices,IBMResearch25July202216/104EfficientEfficientNeuralArchitectureSearchOne-ShotArchitectureSearchOverviewn exp(αi,j))o∈O之o\ exp(αi,j))o∈O之o\∈Oexp(α)(3)byandnetworkweights?Magnitudeofanoperation:對oi\,j))?Derivediscretearchitectureby(1)o(i,j)=argmaxo∈Oαi,j)(2)HanxiaoLiu,KarenSimonyan,andYimingYang.“DARTS:DifferentiableArchitectureSearch”.In:ProceedingsoftheInternationalConferenceonLearningRepresentations,ICLR2019,NewOrleans,Louisiana,USA.2019MartinMartinWistuba,TejaswiniPedapati,AmazonWebServices,IBMResearch25July202217/104111222++++333EfficientNeuralArchitectureSearchOne-ShotArchitectureSearchOverview111222++++333EfficientNeuralArchitectureSearchOne-ShotArchitectureSearchOverview112233αAL(mα(G)L‘mα(β),θ,dtrain,+R(θ),dvalid)(4)MartinMartinWistuba,TejaswiniPedapati,AmazonWebServices,IBMResearch25July202218/104111222++++333EfficientNeuralArchitectureSearchOne-ShotArchitectureSearchOverview111222++++333EfficientNeuralArchitectureSearchOne-ShotArchitectureSearchOverview112233αAL(mα(G)L‘mα(β),θ,dtrain,+R(θ),dvalid)(4)MartinMartinWistuba,TejaswiniPedapati,AmazonWebServices,IBMResearch25July202218/104111222++++333EfficientNeuralArchitectureSearchOne-ShotArchitectureSearchOverview111222++++333EfficientNeuralArchitectureSearchOne-ShotArchitectureSearchOverview112233αAL(mα(G)L‘mα(β),θ,dtrain,+R(θ),dvalid)(4)MartinMartinWistuba,TejaswiniPedapati,AmazonWebServices,IBMResearch25July202218/104111222++++333EfficientNeuralArchitectureSearchOne-ShotArchitectureSearchOverview111222++++333EfficientNeuralArchitectureSearchOne-ShotArchitectureSearchOverview112233αAL(mα(G)L‘mα(β),θ,dtrain,+R(θ),dvalid)(4)MartinMartinWistuba,TejaswiniPedapati,AmazonWebServices,IBMResearch25July202218/10411 2++233EfficientNeuralArchitectureSearch11 2++233EfficientNeuralArchitectureSearchOne-ShotArchitectureSearchOverview11 o2+o2+ oo3αAL(mα(G)L‘mα(β),θ,dtrain,+R(θ),dvalid)(4)MartinMartinWistuba,TejaswiniPedapati,AmazonWebServices,IBMResearch25July202218/104EfficientEfficientNeuralArchitectureSearchOne-ShotArchitectureSearchOverviewonnLval(θ?(α),α)(5)s.t.θ?(α)=argminθLtrain(θ,α)(6)Algorithm2DARTS–DifferentiableArchitectureSearchInput:Amixedoperation(i,j)3:?αLval(θ?ξ?θLtrain(θ,α),α)4:(ξ=0ifusingfirst-orderapproximation)5:Updateweightsθbydescending?θLtrain(θ,α)MartinMartinWistuba,TejaswiniPedapati,AmazonWebServices,IBMResearch25July202219/104EfficientEfficientNeuralArchitectureSearchOne-ShotArchitectureSearchShortcomings1.IntroductionMemoryConsumptionTSCollapseDARTSDiscretizationncy2.3OtherflavoursofOne-shotNAS3.Zero-ShotNASLearningforNAS 25July202220/104EfficientEfficientNeuralArchitectureSearchOne-ShotArchitectureSearchShortcomingsMartinMartinWistuba,TejaswiniPedapati,AmazonWebServices,IBMResearch25July202220/104EfficientEfficientNeuralArchitectureSearchOne-ShotArchitectureSearchShortcomings1.IntroductionMemoryConsumptionTSCollapseDARTSDiscretizationncy2.3OtherflavoursofOne-shotNAS3.Zero-ShotNASLearningforNAS 25July202221/104EfficientEfficientNeuralArchitectureSearchOne-ShotArchitectureSearchShortcomingsnedgeisaweightedsumofalltheoperations對j)oi(x)lMartinMartinWistuba,TejaswiniPedapati,AmazonWebServices,IBMResearch25July202221/104EfficientEfficientNeuralArchitectureSearchOne-ShotArchitectureSearchShortcomingsory?InProxylessNASonlyonepath/operationisactiveatatime.g=binarize(p1,···,pN)=〈···withprobabilityp1,(7)withprobabilitypN.N'o1(x)m(x)=工gioi(x)=〈···i=1'(oN(x)withprobabilityp1(8)withprobabilitypN.HanCai,LigengZhu,andSongHan.“ProxylessNAS:DirectNeuralArchitectureSearchonTargetTaskandHardware”.In:ProceedingsoftheInternationalConferenceonLearningRepresentations,ICLR2019,NewOrleans,Louisiana,USA.2019MartinMartinWistuba,TejaswiniPedapati,AmazonWebServices,IBMResearch25July202222/104ShortcomingsEfficientNeuralArchitectureSearchOne-ShotArchitectureSearchShortcomings INPUTINPUTCVCVCVCVIdentity...PLPeimhtrsCVCVIdentity...PL αβσ…δ αβσ…δArchitectureParametersαβσ…δ 100…0BinaryGate(0:prune,1:keep)010…0 OUTPUTfmapinmemoryOUTPUT(1)Updateweightparametersfmapnotinmemory(2)UpdatearchitectureparameterssmemoryatanypointMartinMartinWistuba,TejaswiniPedapati,AmazonWebServices,IBMResearch25July202223/104 EfficientNeuralArchitectureSearchOne-ShotArchitectureSearchShortcomingsPCDARTSoffYuhuiXuetal.“PC-DARTS:PartialChannelConnectionsforMemory-EfficientArchitectureSearch”.In:8thInternationalConferenceonLearningRepresentations,ICLR2020,AddisAbaba,Ethiopia,April26-30,2020.2020MartinMartinWistuba,TejaswiniPedapati,AmazonWebServices,IBMResearch25July202224/104ShortcomingsEfficientNeuralArchitectureSearchOne-ShotArchitectureSearchShortcomings exp{αj}fiC(xi;Si,j)=對o\∈Oexp{α}·o(Si,j*xi)+(1?Si,j)* exp{αj}x=對ifi,j(xi).(10)orepochsMartinMartinWistuba,TejaswiniPedapati,AmazonWebServices,IBMResearch25July202225/104EfficientEfficientNeuralArchitectureSearchOne-ShotArchitectureSearchShortcomings1.IntroductionMemoryConsumptionTSCollapseHYPERLINK\l"_bo