深層神經(jīng)網(wǎng)絡(luò)課件

為什么應(yīng)用深度學(xué)習(xí)？為什么應(yīng)用深度學(xué)習(xí)？？為什么應(yīng)用深度學(xué)習(xí)？為什么應(yīng)用深度學(xué)習(xí)？為什么應(yīng)用深度學(xué)習(xí)？為什么應(yīng)用深度學(xué)習(xí)？深度學(xué)習(xí)基礎(chǔ)深度學(xué)習(xí)基礎(chǔ)深度學(xué)習(xí)基礎(chǔ)深度學(xué)習(xí)基礎(chǔ)深度學(xué)習(xí)基礎(chǔ)深度學(xué)習(xí)基礎(chǔ)深度學(xué)習(xí)基礎(chǔ)深度學(xué)習(xí)基礎(chǔ)深度學(xué)習(xí)基礎(chǔ)深度學(xué)習(xí)基礎(chǔ)深度學(xué)習(xí)基礎(chǔ)深度學(xué)習(xí)基礎(chǔ)深度學(xué)習(xí)基礎(chǔ)深度學(xué)習(xí)基礎(chǔ)深度學(xué)習(xí)基礎(chǔ)深度學(xué)習(xí)基礎(chǔ)深度學(xué)習(xí)基礎(chǔ)深度學(xué)習(xí)基礎(chǔ)深度學(xué)習(xí)基礎(chǔ)深度學(xué)習(xí)應(yīng)用計(jì)算機(jī)視覺(jué)深度學(xué)習(xí)應(yīng)用深度學(xué)習(xí)應(yīng)用深度學(xué)習(xí)應(yīng)用深度學(xué)習(xí)應(yīng)用深度學(xué)習(xí)應(yīng)用深度學(xué)習(xí)應(yīng)用自然語(yǔ)言處理詞嵌入情感分析主題模型神經(jīng)翻譯閱讀理解…..常用深度學(xué)習(xí)網(wǎng)絡(luò)模型和算法

RBM網(wǎng)絡(luò)DBN網(wǎng)絡(luò)CNN網(wǎng)絡(luò)RNN網(wǎng)絡(luò)GAN網(wǎng)絡(luò)AE網(wǎng)絡(luò)深度神經(jīng)網(wǎng)絡(luò)基礎(chǔ)及應(yīng)用342012年，由人工智能和機(jī)器學(xué)習(xí)頂級(jí)學(xué)者AndrewNg和分布式系統(tǒng)頂級(jí)專家JeffDean領(lǐng)銜的夢(mèng)幻陣容，開(kāi)始打造GoogleBrain項(xiàng)目，用包含16000個(gè)CPU核的并行計(jì)算平臺(tái)訓(xùn)練超過(guò)10億個(gè)神經(jīng)元的深度神經(jīng)網(wǎng)絡(luò)，在語(yǔ)音識(shí)別和圖像識(shí)別等領(lǐng)域取得了突破性的進(jìn)展。該系統(tǒng)通過(guò)分析YouTube上選取的視頻，采用無(wú)監(jiān)督的方式訓(xùn)練深度神經(jīng)網(wǎng)絡(luò)，可將圖像自動(dòng)聚類。在系統(tǒng)中輸入“cat”后，結(jié)果在沒(méi)有外界干涉的條件下，識(shí)別出了貓臉。GoogleBrain深度神經(jīng)網(wǎng)絡(luò)基礎(chǔ)及應(yīng)用352012年，微軟首席研究官RickRashid在21世紀(jì)的計(jì)算大會(huì)上演示了一套自動(dòng)同聲傳譯系統(tǒng)[1]，將他的英文演講實(shí)時(shí)轉(zhuǎn)換成與他音色相近、字正腔圓的中文演講。同聲傳譯需要經(jīng)歷語(yǔ)音識(shí)別、機(jī)器翻譯、語(yǔ)音合成三個(gè)步驟。該系統(tǒng)一氣呵成，流暢的效果贏得了一致認(rèn)可，深度學(xué)習(xí)則是這一系統(tǒng)中的關(guān)鍵技術(shù)。微軟語(yǔ)音識(shí)別[1]RickRashid,SpeechRecognitionBreakthroughfortheSpoken,TranslatedWord/watch?v=Nu-nlQqFCKg深度神經(jīng)網(wǎng)絡(luò)基礎(chǔ)及應(yīng)用36深度學(xué)習(xí)用于視頻分析（起步階段）最直接的做法是將視頻視為三維圖像，直接應(yīng)用卷積網(wǎng)絡(luò)[1]，在每一層學(xué)習(xí)三維濾波器，沒(méi)有考慮到時(shí)間維和空間維的差異性。另外一種簡(jiǎn)單但更加有效的思路是通過(guò)預(yù)處理計(jì)算光流場(chǎng)，作為卷積網(wǎng)絡(luò)的一個(gè)輸入通道[2]。如右圖。傳統(tǒng)的方法大多采用線性動(dòng)態(tài)系統(tǒng)建模。在一些最新的研究工作中[3]，長(zhǎng)短記憶網(wǎng)絡(luò)（LSTM）正在受到廣泛關(guān)注，它可以捕捉長(zhǎng)期依賴性，對(duì)視頻中復(fù)雜的動(dòng)態(tài)建模。[1]S.Ji,W.Xu,M.Yang,andK.Yu.3dconvolutionalneuralnetworksforhumanactionrecognition.IEEETrans.onPatternAnalysisandMachineIntelligence,35(1):221–231,2013.

[2]K.SimonyanandA.Zisserman.Two‐StreamConvolutionalNetworksforActionRecognitioninVideos.arXiv:1406.2199,2014.

[3]J.Donahue,L.A.Hendricks,S.Guadarrama,M.Rohrbach,S.Venugopalan,K.Saenko,andT.Darrell.Long‐termrecurrentconvolutionalnetworksforvisualrecognitionanddescription.arXiv:1411.4389,2014.

Bishop,C.M.(2006).PatternRecognitionandMachineLearning(InformationScienceandStatistics).Springer-VerlagNewYork,Inc.Bengio,Y.(2009).LearningDeepArchitecturesforAI.Foundations&TrendsinMachineLearning,2,1-127.Bengio,Y.(2015).DeepLearning.官網(wǎng)電子書(shū)鏈接：

http://goodfeli.github.io/dlbook/深度學(xué)習(xí)網(wǎng)站（software、demos、datasets、blog…）/UFLDL教程：/wiki/index.php/UFLDL教程Onlinebook：NeuralNetworkandDeepLearning/index.html常用深度學(xué)習(xí)網(wǎng)絡(luò)模型和算法

RBM網(wǎng)絡(luò)DBN網(wǎng)絡(luò)CNN網(wǎng)絡(luò)RNN網(wǎng)絡(luò)GAN網(wǎng)絡(luò)AE網(wǎng)絡(luò)CodeInputPrediction(s)Error(s)Wewant

thecodes

torepresent

theinputs

inthedataset.Thecodeshould

beacompact

representation

oftheinputs:

low-dimensional

and/orsparse.CodeInputTarget

=inputCodeInput“Bottleneck”code

i.e.,low-dimensional,

typicallydense,

distributed

representation“Overcomplete”code

i.e.,high-dimensional,

alwayssparse,

distributed

representationTarget

=inputCodeInputCode

predictionEncoding

“energy”Decoding

“energy”Input

decodingEncodingenergyDecodingenergyEncodingenergyDecodingenergyForonesampletForallTsamplesHowdowegetthecodesZ?coefficientoftheencodererrorWenoteW={C,bC,D,bD}Learntheparameters(weights)W

oftheencoderanddecoder

giventhecurrentcodesZInferthe

codesZ

giventhecurrent

modelparametersWRelationshiptoExpectation-Maximization

Takeagradientdescentstep

ontheparameters(weights)W

oftheencoderanddecoder

giventhecurrentcodesZIteratedgradientdescent(?)

onthe

code

Z(t)

giventhecurrent

modelparametersWRelationshiptoGeneralizedEM

Expectation-MaximizationEnergyofinputsandcodesInputdatalikelihoodMaximumAPosteriori:takeminimalenergycodeZDonotmarginalizeover:

takemaximumlikelihood

latentcodeinsteadEnforcesparsityonZ

toconstrainZand

avoidcomputing

partitionfunctionCodeInputCode

predictionEncoding

energyDecoding

energyInput

decodingCodeInputCode

predictionEncoding

energy[Ranzato,Boureau&LeCun,“SparseFeatureLearningforDeepBeliefNetworks”,NIPS,2007]CodeInputCode

predictionEncoding

energy[Ranzato,Boureau&LeCun,“SparseFeatureLearningforDeepBeliefNetworks”,NIPS,2007]SparsecodingOvercompletecodeInputDecoding

errorInput

decodingSparsityconstraint[Olshausen&Field,“Sparsecodingwithanovercompletebasisset:astrategyemployedbyV1?”,VisionResearch,1997]Decoding

errorInput

decodingSparsityconstraintInput[Olshausen&Field,“Sparsecodingwithanovercompletebasisset:astrategyemployedbyV1?”,VisionResearch,1997]OvercompletecodeSparsecodingLimitationsofsparsecodingAtruntime,assumingatrainedmodelW,

inferringthecodeZgivenaninputsampleX

isexpensiveNeedatweakonthemodelweightsW:

normalizethecolumnsofWtounitlength

aftereachlearningstepOtherwise:codepulledto0

bysparsityconstraintweightsgoto

infinitytocompensateCodeInputCode

predictionCode

errorDecoding

errorInput

decodingSparsityconstraint[Ranzato,Poultney,Chopra&LeCun,“EfficientLearningofSparseRepresentationswithanEnergy-BasedModel”,NIPS,2006;

Ranzato,Boureau&LeCun,“SparseFeatureLearningforDeepBeliefNetworks”,NIPS,2007]SparseAuto-encoderSymmetricsparseauto-encoderCodeCode

predictionCode

errorDecoding

errorInput

decodingSparsityconstraintInput[Ranzato,Poultney,Chopra&LeCun,“EfficientLearningofSparseRepresentationswithanEnergy-BasedModel”,NIPS,2006;

Ranzato,Boureau&LeCun,“SparseFeatureLearningforDeepBeliefNetworks”,NIPS,2007]EncodermatrixW

issymmetricto

decodermatrixWTSemi-supervisedlearningofauto-encodersAddclassifiermoduletothecodesWhenainputX(t)hasalabelY(t),

back-propagatethepredictionerroronY(t)

tothecodeZ(t)StacktheencodersTrainlayer-wise[Ranzato&Szummer,“Semi-supervisedlearningofcompactdocumentrepresentationswithdeepnetworks”,ICML,2008;

Mirowski,Ranzato&LeCun,“Dynamicauto-encodersforsemanticindexing”,NIPSDeepLearningWorkshop,2010]y(t)y(t+1)z(1)(t)z(1)(t+1)documentclassifierf1x(t)x(t+1)y(t)y(t+1)z(2)(t)z(2)(t+1)documentclassifierf2y(t)y(t+1)z(3)(t)z(3)(t+1)documentclassifierf3auto-encoderg3,h3auto-encoderg2,h2auto-encoderg1,h1RandomwalkwordhistogramsWeknowitisgoodtolearnasmallmodel.Fromthisfullyconnectedmodel,dowereallyneedalltheedges?Cansomeofthesebeshared?CanrepresentasmallregionwithfewerparametersAfilterACNNisaneuralnetworkwithsomeconvolutionallayers(andsomeotherlayers).Aconvolutionallayerhasanumberoffiltersthatdoesconvolutionaloperation.Beakdetector1000010100100011001000100100100010106x6image1-1-1-11-1-1-11Filter1-11-1-11-1-11-1Filter2……Thesearethenetworkparameterstobelearned.Eachfilterdetectsasmallpattern(3x3).1000010100100011001000100100100010106x6image1-1-1-11-1-1-11Filter13-1stride=1Dotproduct1000010100100011001000100100100010106x6image1-1-1-11-1-1-11Filter13-3Ifstride=21000010100100011001000100100100010106x6image1-1-1-11-1-1-11Filter13-1-3-1-310-3-3-3013-2-2-1stride=11000010100100011001000100100100010106x6image3-1-3-1-310-3-3-3013-2-2-1-11-1-11-1-11-1Filter2-1-1-1-1-1-1-21-1-1-21-10-43Repeatthisforeachfilterstride=1Two4x4imagesForming2x4x4matrixFeatureMapColorimage:RGB3channels1000010100100011001000100100100010101000010100100011001000100100100010101000010100100011001000100100100010101-1-1-11-1-1-11Filter1-11-1-11-1-11-1Filter21-1-1-11-1-1-111-1-1-11-1-1-11-11-1-11-1-11-1-11-1-11-1-11-1Colorimage100001010010001100100010010010001010imageconvolution-11-1-11-1-11-11-1-1-11-1-1-11…………100001010010001100100010010010001010Convolutionv.s.FullyConnectedFully-connected1000010100100011001000100100100010106x6image1-1-1-11-1-1-11Filter1123…89…131415…Onlyconnectto9inputs,notfullyconnected4:10:161000010000113fewerparameters!1000010100100011001000100100100010101-1-1-11-1-1-11Filter11:2:3:…7:8:9:…13:14:15:…4:10:16:1000010000113-1Sharedweights6x6imageFewerparametersEvenfewerparameters3-1-3-1-310-3-3-3013-2-2-1-1-1-1-1-1-1-21-1-1-21-10-43WhyPoolingSubsamplingpixelswillnotchangetheobjectSubsamplingbirdbirdWecansubsamplethepixelstomakeimagesmallerfewerparameterstocharacterizetheimageACNNcompressesafullyconnectednetworkintwoways:ReducingnumberofconnectionsSharedweightsonthee