版權(quán)說明:本文檔由用戶提供并上傳,收益歸屬內(nèi)容提供方,若內(nèi)容存在侵權(quán),請(qǐng)進(jìn)行舉報(bào)或認(rèn)領(lǐng)
文檔簡介
DeepLearningforNaturalLanguageProcessingJonathanMugan,PhDNLPCommunityDayJune4,2015OverviewAboutmeandDeepGrammar(4minutes)IntroductiontoDeepLearningforNLPRecurrentNeuralNetworksDeepLearningandQuestionAnsweringLimitationsofDeepLearningforNLPHowYouCanGetStartedTheimportanceoffindingdumbmistakesTheimportanceoffindingdumbmistakesOverviewAboutmeandDeepGrammar(4minutes)IntroductiontoDeepLearningforNLPRecurrentNeuralNetworksDeepLearningandQuestionAnsweringLimitationsofDeepLearningforNLPHowYouCanGetStartedOverviewAboutmeandDeepGrammar(4minutes)IntroductiontoDeepLearningforNLPRecurrentNeuralNetworksDeepLearningandQuestionAnsweringLimitationsofDeepLearningforNLPHowYouCanGetStartedDeeplearningenablessub-symbolicprocessingSymbolicsystemscanbebrittle.Iboughtacar.<i><bought><a><car><.>Youhavetoremembertorepresent“purchased”and“automobile.”Whatabout“truck”?Howdoyouencodethemeaningoftheentiresentence?DeeplearningbeginswithalittlefunctionItallstartswithahumblelinearfunctioncalledaperceptron.weight1?input1weight2?input2weight3?input3sum?Perceptron:Ifsum>threshold:output1Else:output0Example:Theinputscanbeyourdata.Question:ShouldIbuythiscar?0.2?gasmilage0.3?horepower0.5?numcupholderssum?Perceptron:Ifsum>threshold:buyElse:walkTheselittlefunctionsarechainedtogetherDeeplearningcomesfromchainingabunchoftheselittlefunctionstogether.Chainedtogether,theyarecalledneurons.whereTocreateaneuron,weaddanonlinearitytotheperceptrontogetextrarepresentationalpowerwhenwechainthemtogether.Ournonlinearperceptronis
sometimescalledasigmoid.
PlotofasigmoidSingleartificialneuronOutput,orinputto
nextneuronweight1?input1weight2?input2weight3?input3Three-layeredneuralnetworkAbunchofneuronschainedtogetheriscalledaneuralnetwork.Layer2:hiddenlayer.Called
thisbecauseitisneitherinput
noroutput.Layer3:output.E.g.,cat
ornotacat;buythecaror
walk.Layer1:inputdata.Can
bepixelvaluesorthenumber
ofcupholders.Thisnetworkhasthreelayers.(Someedgeslighter
toavoidclutter.)[16.2,17.3,?52.3,11.1]TrainingwithsupervisedlearningSupervisedLearning:Youshowthenetworkabunchofthingswithalabelssayingwhattheyare,andyouwantthenetworktolearntoclassifyfuturethingswithoutlabels.Example:herearesomepicturesofcats.Tellmewhichoftheseotherpicturesareofcats.Totrainthenetwork,wanttofindtheweightsthatcorrectlyclassifyallofthetrainingexamples.Youhopeitwillworkonthetestingexamples.DonewithanalgorithmcalledBackpropagation[Rumelhartetal.,1986].[16.2,17.3,?52.3,11.1]TrainingwithsupervisedlearningSupervisedLearning:Youshowthenetworkabunchofthingswithalabelssayingwhattheyare,andyouwantthenetworktolearntoclassifyfuturethingswithoutlabels.
[16.2,17.3,?52.3,11.1]Learningislearningthe
parametervalues.WhyGoogle’sdeeplearningtoolboxiscalledTensorFlow.DeeplearningisaddingmorelayersThereisnoexactdefinitionof
whatconstitutes“deeplearning.”
Thenumberofweights(parameters)isgenerallylarge.Somenetworkshavemillionsofparametersthatarelearned.(Someedgesomitted
toavoidclutter.)[16.2,17.3,?52.3,11.1]RecallourstandardarchitectureLayer2:hiddenlayer.Called
thisbecauseitisneitherinput
noroutput.Layer3:output.E.g.,cat
ornotacat;buythecaror
walk.Layer1:inputdata.Can
bepixelvaluesorthenumber
ofcupholders.Isthisacat?[16.2,17.3,?52.3,11.1]NeuralnetswithmultipleoutputsOkay,butwhatkindofcatisit?
Introduceanewnode
calledasoftmax.Probabilitya
housecatProbabilitya
lionProbabilitya
pantherProbabilitya
bobcatJustnormalizetheoutputoverthesumoftheotheroutputs(usingtheexponential).Givesaprobability.[16.2,17.3,?52.3,11.1]Learningwordvectors
Fromthesentence,“Themanranfast.”
Probability
of“fast”Probability
of“slow”Probability
of“taco”Probability
of“bobcat”Learnsavectorforeachwordbasedonthe“meaning”inthesentenceby
tryingtopredictthenextword[Bengioetal.,2003].Thesenumbersupdatedalongwiththeweightsandethevectorrepresentationsofthewords.Comparingvectorandsymbolicrepresentations
Vectorshaveasimilarityscore.Atacoisnotaburritobutsimilar.Symbolscanbethesameornot.AtacoisjustasdifferentfromaburritoasaToyota.Vectorshaveinternalstructure[Mikolovetal.,2013].Italy–Rome=France–ParisKing–Queen=Man–WomanSymbolshavenostructure.Symbolsarearbitrarilyassigned.Meaningrelativetoothersymbols.Vectorsaregroundedinexperience.Meaningrelativetopredictions.Abilitytolearnrepresentationsmakesagentslessbrittle.OverviewAboutmeandDeepGrammar(4minutes)IntroductiontoDeepLearningforNLPRecurrentNeuralNetworksDeepLearningandQuestionAnsweringLimitationsofDeepLearningforNLPHowYouCanGetStartedOverviewAboutmeandDeepGrammar(4minutes)IntroductiontoDeepLearningforNLPRecurrentNeuralNetworksDeepLearningandQuestionAnsweringLimitationsofDeepLearningforNLPHowYouCanGetStartedEncodingsentencemeaningintoavectorh0The“Thepatientfell.”Encodingsentencemeaningintoavectorh0Theh1patient“Thepatientfell.”Encodingsentencemeaningintoavectorh0Theh1patienth2fell“Thepatientfell.”EncodingsentencemeaningintoavectorLikeahiddenMarkovmodel,butdoesn’tmaketheMarkovassumptionandbenefitsfromavectorrepresentation.h0Theh1patienth2fellh3.“Thepatientfell.”DecodingsentencemeaningMachinetranslation,orstructurelearningmoregenerally.
Elh3DecodingsentencemeaningMachinetranslation,orstructurelearningmoregenerally.
Elh3h4DecodingsentencemeaningMachinetranslation,orstructurelearningmoregenerally.
Elh3pacienteh4DecodingsentencemeaningMachinetranslation,orstructurelearningmoregenerally.
Elh3pacienteh4cayóh5.h5[Choetal.,2014]Itkeepsgeneratinguntilitgeneratesastopsymbol.GeneratingimagecaptionsConvolutionalneuralnetworkAnh0angryh1sisterh2.h3[KarpathyandFei-Fei,2015][Vinyalsetal.,2015]Imagecaptionexamples[KarpathyandFei-Fei,2015]See:Attention[Bahdanauetal.,2014]Elh3pacienteh4cayóh5.h5h0Theh1patienth2fellh3.RNNsandStructureLearningThesearesometimescalledseq2seqmodels.Inadditiontomachinetranslationandgeneratingcaptionsforimages,canbeusedtolearnjustaboutanykindofstructureyou’dwant,aslongasyouhavelotsoftrainingdata.OverviewAboutmeandDeepGrammar(4minutes)IntroductiontoDeepLearningforNLPRecurrentNeuralNetworksDeepLearningandQuestionAnsweringLimitationsofDeepLearningforNLPHowYouCanGetStartedOverviewAboutmeandDeepGrammar(4minutes)IntroductiontoDeepLearningforNLPRecurrentNeuralNetworksDeepLearningandQuestionAnsweringLimitationsofDeepLearningforNLPHowYouCanGetStartedDeeplearningandquestionansweringRNNsanswerquestions.WhatisthetranslationofthisphrasetoFrench?Whatisthenextword?Attentionisusefulforquestionanswering.Thiscanbegeneralizedtowhichfactsthelearnershouldpayattentiontowhenansweringquestions.DeeplearningandquestionansweringBobwenthome.Timwenttothejunkyard.Bobpickedupthejar.Bobwenttotown.Whereisthejar?A:townMemoryNetworks[Westonetal.,2014]Updatesmemoryvectorsbasedonaquestionandfindsthebestonetogivetheoutput.Theofficeisnorthoftheyard.Thebathisnorthoftheoffice.Theyardiswestofthekitchen.Howdoyougofromtheofficetothekitchen?A:south,eastNeuralReasoner[Pengetal.,2015]Encodesthequestionandfactsinmanylayers,andthefinallayerisputthroughafunctionthatgivestheanswer.OverviewAboutmeandDeepGrammar(4minutes)IntroductiontoDeepLearningforNLPRecurrentNeuralNetworksDeepLearningandQuestionAnsweringLimitationsofDeepLearningforNLPHowYouCanGetStartedOverviewAboutmeandDeepGrammar(4minutes)IntroductiontoDeepLearningforNLPRecurrentNeuralNetworksDeepLearningandQuestionAnsweringLimitationsofDeepLearningforNLPHowYouCanGetStartedLimitationsofdeeplearningTheencodedmeaningisgroundedwithrespecttootherwords.Thereisnolinkagetothephysicalworld."ICubLugan01Reaching".LicensedunderCCBY-SA3.0viaWikipedia-TheiCubLimitationsofdeeplearningBobwenthome.Timwenttothejunkyard.Bobpickedupthejar.Bobwenttotown.Whereisthejar?A:townDeeplearninghasnounderstandingofwhatitmeansforthejartobeintown.Forexamplethatitcan’talsobeatthejunkyard.OrthatitmaybeinBob’scar,orstillinhishands.Theencodedmeaningisgroundedwithrespecttootherwords.Thereisnolinkagetothephysicalworld.LimitationsofdeeplearningImagineadudestandingonatable.Howwouldacomputerknowthatifyoumovethetableyoualsomovethedude?Likewise,howcouldacomputerknowthatitonlyrainsoutside?Or,asMarvinMinskyasks,howcouldacomputerlearnthatyoucanpullaboxwithastringbutnotpushit?LimitationsofdeeplearningImagineadudestandingonatable.Howwouldacomputerknowthatifyoumovethetableyoualsomovethedude?Likewise,howcouldacomputerknowthatitonlyrainsoutside?Or,asMarvinMinskyasks,howcouldacomputerlearnthatyoucanpullaboxwithastringbutnotpushit?Nooneknowshowtoexplainallofthesesituationstoacompute
溫馨提示
- 1. 本站所有資源如無特殊說明,都需要本地電腦安裝OFFICE2007和PDF閱讀器。圖紙軟件為CAD,CAXA,PROE,UG,SolidWorks等.壓縮文件請(qǐng)下載最新的WinRAR軟件解壓。
- 2. 本站的文檔不包含任何第三方提供的附件圖紙等,如果需要附件,請(qǐng)聯(lián)系上傳者。文件的所有權(quán)益歸上傳用戶所有。
- 3. 本站RAR壓縮包中若帶圖紙,網(wǎng)頁內(nèi)容里面會(huì)有圖紙預(yù)覽,若沒有圖紙預(yù)覽就沒有圖紙。
- 4. 未經(jīng)權(quán)益所有人同意不得將文件中的內(nèi)容挪作商業(yè)或盈利用途。
- 5. 人人文庫網(wǎng)僅提供信息存儲(chǔ)空間,僅對(duì)用戶上傳內(nèi)容的表現(xiàn)方式做保護(hù)處理,對(duì)用戶上傳分享的文檔內(nèi)容本身不做任何修改或編輯,并不能對(duì)任何下載內(nèi)容負(fù)責(zé)。
- 6. 下載文件中如有侵權(quán)或不適當(dāng)內(nèi)容,請(qǐng)與我們聯(lián)系,我們立即糾正。
- 7. 本站不保證下載資源的準(zhǔn)確性、安全性和完整性, 同時(shí)也不承擔(dān)用戶因使用這些下載資源對(duì)自己和他人造成任何形式的傷害或損失。
最新文檔
- 統(tǒng)編版語文二年級(jí)上冊(cè)課內(nèi)閱讀(PDF版 含答案)
- 《2024年 新發(fā)展理念與文化自信》范文
- 有的人課文課件
- 八年級(jí)下學(xué)期期末考試語文試題(PDF版含答案)
- 八年級(jí)上學(xué)期1月期末考試語文試題(PDF版含答案)
- 修建合同模板
- 《2024年 寫作學(xué)習(xí)環(huán)境的建構(gòu)-活動(dòng)理論的視角》范文
- 《 基于SOPC的聲紋識(shí)別系統(tǒng)的設(shè)計(jì)與現(xiàn)》范文
- 《2024年 卡介菌多糖核酸對(duì)慢性蕁麻疹患者外周血淋巴細(xì)胞Th1-Th2分化的影響》范文
- 《2024年 Fe3O4磁性納米材料的制備及水處理應(yīng)用進(jìn)展》范文
- 【廣州浪奇公司存貨內(nèi)部控制缺陷的案例探析8100字(論文)】
- 勞動(dòng)主題《了解志愿服務(wù)》(教學(xué)設(shè)計(jì))-二年級(jí)勞動(dòng)奧教版
- DZ∕T 0173-2022 大地電磁測(cè)深法技術(shù)規(guī)程(正式版)
- 人工智能技術(shù)調(diào)研報(bào)告
- 部隊(duì)物業(yè)管理方案
- JT-T-331.4-1996港口碼頭勞動(dòng)定員標(biāo)準(zhǔn)集裝箱碼頭-PDF解密
- 中小學(xué)生中醫(yī)藥科普知識(shí)競賽
- 《紙質(zhì)文物修復(fù)與保護(hù)》課件-31古籍書冊(cè)結(jié)構(gòu)
- 30道資產(chǎn)管理經(jīng)理崗位常見面試問題含HR問題考察點(diǎn)及參考回答
- 【第二講】大學(xué)生就業(yè)指導(dǎo)之職業(yè)素養(yǎng)與職業(yè)能力課件
- 中學(xué)風(fēng)險(xiǎn)辨識(shí)評(píng)估和應(yīng)急資源調(diào)查報(bào)告
評(píng)論
0/150
提交評(píng)論