深度學(xué)習(xí)一線實(shí)戰(zhàn)暑期研討基礎(chǔ)

234

DiscreteValueContinuousValue圖像表示：Gabor圖像表示：Gabor圖像集表示：Manifold,GMM,LMNN,NCA…..詞典學(xué)習(xí)&CreditedtoProf.XiaogangWangandProf.Shiguang CreditedtoProf.Songchun DiscreteValueContinuousValue端到端學(xué)習(xí)(EndtoCreditedtoProf.Shiguang CreditedtoDr.Naiyan CreditedtoDr.Naiyan 多多PascalVOC目標(biāo)檢測(cè)CreditedtoProf.ShiguangShanwith

Fastdescriptorcoding(LLCJinjunWang’sCVPR10&Super+SVMLin,Y’sCompressedFisherVectorsPerronnin,F’sECCV10,Sanchez,J’sDCNNAlexNet8層網(wǎng)絡(luò)KrizhevskyA’sDCNN:基于網(wǎng)絡(luò)可視化技術(shù)Zeiler,M.D’sECCV14在AlexNetDCNN:NetChristianSzegedy’sCVPR15，22層+Inception結(jié)DCNN:VGGNetKarenSimonyan’sarXiv14，19DCNN:ResidualNetKaimingHe’sarXiv15,152SelectiveSearchJ.R.R.Uijlings’sIJCV13+efficientencodingdenselysampledcolordescriptorsvandeSande’sTPAMI10,CVPR14RCNNRossGirshick’sCVPR14+NetworkInNetworkMinLin’sFasterRCNNShaoqingRen’sNIPS15DeepResidualNetworkKaimingHe’sMeanMean

CombinedMultishot

VectorVector

gh-dimintgh-dim

TLTL

Unrestricted Outside Unrestricted+LabeledOutside 網(wǎng)絡(luò)變大變深（VGGFace16FaceNet22層數(shù)據(jù)量不斷增大（DeepFace400萬，F(xiàn)aceNet2億3DeepID[Sun’sDeepID2[Sun’sDeepID2+[Sun’s-1FaceNet[Schroff’s1ModifiedfromProf.ModifiedfromProf.LeCunandProf. 來自：BoleiZhouetalObjectDetectorsEmergeinDeepSceneCNNsICLRConv，Conv，Pooling,ReLU,connected,BN，NIN， N,SiameseNet,機(jī)制LossFunc,e.g.SoftmaxLoss,SigmoidCross-entropy,…BP,SGD,AdaDelta,Path-Dropout,Fine-tune,CreditedtoProf.MeinaKan,PHDStudentXinLiuandShuzhe McCulloch,Warren;WalterPitts(1943)."ALogicalCalculusofIdeasImmanentinNervousActivity".BulletinofMathematicalBiophysics5(4):115–133F.Rosenblatt.Theperceptron:Aprobabilisticmodelforinformationstorageandorganizationinthebrain. PsychologicalReview,65:386-408,Minsky&Papert的專著Perceptron(1969) Novikoff,A.B.J.(1962).Onconvergenceproofsonperceptrons.ProceedingsoftheSymposiumontheMathematicalTheoryofAutomata(pp.615–622)IEEEFrankRosenblatt2014-GeoffreyE.Hinton2012-VladimirN.Vapnik2009-JohnJ.Hopfield多層感知機(jī)卷積網(wǎng)絡(luò)

DecisionSparseGraphRumelhart,DavidE.;Hinton,GeoffreyE.;Williams,RonaldJ.(8October1986)."Learningrepresentationsbyback-propagatingerrors".Nature323(6088):533–536.CreditedtoProf.ShiguangHinton,G.E.,Osindero,S.andTeh,Y.,Afastlearningalgorithmfordeepbeliefnets.NeuralComputation18:1527-1554,2006Hinton,G.E.andSalakhutdinov,R.R.Reducingthedimensionalityofdatawithneuralnetworks.Science,Vol.313.no.5786,pp.504-507,28July2006YoshuaBengio,PascalLamblin,DanPopoviciandHugoLaroce,GreedyLayer-WiseTrainingofDeepNetworks,NIPS2006CreditedtoProf.Eric deeplearningDCNNAlexKrizhevsky’s止過擬合,LocalResponseNormalization增強(qiáng)泛化能力評(píng)價(jià)評(píng)價(jià) 評(píng)價(jià)RNNLanguage評(píng)價(jià)CreditedtoProf.MeinaKan,PHDStudentXinLiuandShuzhe ReLU/

MoonMoonStart...：... ，其中??∈?????1,??∈?????1,??∈??：...Backward運(yùn)算：????=???????，????=?? 連續(xù)卷積???,

=?????(??,??)

????,?????????,?????離散卷積????,Caffe實(shí)現(xiàn)：????

=?????(??,??) ??????,?????????,???=?????(??,??) ??????+??,??+??????,多個(gè)FeatureMap: 接

CreditedtoDr.ShaoxinCreditedtoDr.ShaoxinPleasealsoreferto“NotesonConvolutionalNeural …Image:??×??×…

FeatureMatrixX:(????????×????????)×(C×K×…Kernel:????????×??×??×…bias:????????×1×1×

??????+??:????????×(??×??× …………??:????????×(????????×……OutputMatrix:????????×(????????×

Kernelsize:2x2Kernelsize:2x24346378445555686823

S(x)=1/(1+e-xy=xifx>

a(e-1)ifx￡max(w1x1+b1,w2x2+b2 LeCun組2010年的文章Whatisthebestmulti-stagearchitectureforobjectrecognition?嘗試了各種非線性激活函Hinton組的“RectifiedLinearUnitsImproveRestrictedBoltzmannMachines”將NReLU用于RBM訓(xùn)練 48引入Bernoulli隨機(jī)數(shù)????代表dropout測(cè)試階段：DoG.E.Hinton,N.Srivastava,A.Krizhevsky,I.Sutskever,andR.R.Salakhutdinov.Improvingneuralnetworksbypreventingco-adaptationoffeaturedetectors.arXivpreprintarXiv:1207.0580,2012. Batch UntersuchungenzudynamischenneuronalenNetzen(1991LearningLong-TermDependencieswithGradientDescentisDifficult(1994引入三個(gè)Gate結(jié)構(gòu)：ForgetGate,InputGate和Output 狀態(tài)與#1.狀態(tài)“遺忘”控制門forget狀態(tài)與#1.狀態(tài)“遺忘”控制門forget#2.輸入控制門input狀態(tài)與#1.狀態(tài)“遺忘”控制門forget#2.輸入控制門input狀態(tài)與#1.狀態(tài)“遺忘”控制門forget#2.輸入控制門input#3:#4.輸出控制門outputPeepholeGRU（inputgate和forgetgate合并為updateSoftmax+CrossEntropyE=-1N

log( ),l?[0,1,...,K-

n??nkn

CrossEntropyLossH(p,q)=-p(x)logxlog(x)'=x

當(dāng)xSigmoidCrossEntropy

?npnlo1?Tips1:目標(biāo)輸出需要?dú)w一化到[0,1]XinLiu*,ShaoxinLi*etal.AgeNet:DeeplyLearnedRegressorandClassifierforRobustApparentAgeEstimation.InternationalConferenceonComputerVisionChaLearnLaPWorkshop(ICCVW),2015.SoftmaxSigmoidCrossEntropy

E=- E=-

log(?nl),ln?[0,1,...,K-n?n?nSigmoidCrossEntropyLossGroud_truth：Pn=(pn,1-pnSigmoid??,1?

?n)

log?

plo1?

L[ log o1? SoftmaxLoss

Euclidean損失函數(shù)：E=

?n Tips1:默認(rèn)的，Data層和ImageData Tips2:歐式損失前可以增加Sigmoid操 ContrastiveLoss（Siamese

1

[(y)d2+(1-y)max(margin-d,0)2,d

a-

nTips2:Caffe中的例子：TripletBasic適用場(chǎng)景：Learningtorank,人臉識(shí)別F.Schroff,D.Kalenichenko,andJ.Philbin,FaceNet:AUnifiedEmbeddingforFaceRecognition MoonEthanM.Ruddetal,MOON:AMixedObjectiveOptimizationNetworkfortheRecognitionofFacialAttributesCreditedtoProf.Back Rumelhart,DavidE.;Hinton,GeoffreyE.;Williams,RonaldJ.(8October1986)."Learningrepresentationsbyback-propagatingerrors".Nature323(6088):533–536.GradientDescentanditswt+1=wt w

StochasticGradientDescendwt+1=wt wL(w,xi,yi

: =w-1h

L(w,x,y

Mini-batchMini-batchfori=1: WeightL(w)=w)+L(w)= L(wtwt+1=wt+Without WithSGDvs.給定等高線下不同優(yōu)化算法的比 鞍點(diǎn)條件下不同優(yōu)化算法的比Inmathematics,asaddlepointisapointinthe ofafunctionwheretheslopes(derivatives)oforthogonalfunctioncomponentsdefiningthesurfaceezero(astationarypoint)butarenotaloc9以為例，點(diǎn)（0,0）是鞍點(diǎn)，CreditedtoPHDStudentXinLiuandXiaoyi LearningCreditedtoProf.LearningRate在每隔stepsize輪迭代后減少base_lr: :base_lr: :“poly”power:0.5

base_lr:base_lr: :base_lr: :“step” :0.1stepsize:

LR(t)=base_lr·

t)Tbase_lr:0.01 :“inv”base_lr:0.01 :“inv” :power:LR(t)=base_lr·(1+g*iter)-LearningtolearnwithrecurrentneuralMa