cs224d深度學習到自然語言處理

CS224d:DeepNLP

Lecture15:

Applica8onsofDLtoNLP

RichardSocher

richard@metamind.io

Overview

?

Modeloverview:Howtoproperlycomparetoothermodelsand

chooseyourownmodel

?

?

?

?

Wordrepresenta8on

Phrasecomposi8onObjec8vefunc8on

Op8miza8on

?

?

?

?

?

CharacterRNNsontextandcode

MorphologyLogic

Ques8onAnswering

Image–Sentencemapping

Lecture1,Slide2

RichardSocher

5/27/15

Modeloverview:WordVectors

?

Random

?

Word2Vec

?

Glove

?

Dimension–oRendefinesthenumberofmodelparameters

?

Orworkdirectlyoncharactersormorphemes

Lecture1,Slide3

RichardSocher

5/27/15

Modeloverview:PhraseVectorComposi8on

?

Composi8onFunc8ongovernshowexactlywordandphrase

vectorsinteracttocomposemeaning

WVscore

W

s

p

?

Averaging:p=a+b

Lotsofsimplealterna8ves

c1

c2

?

Recursiveneuralnetworks

?

Convolu8onalneuralnetworks

?

Recurrentneuralnetwork

Lecture1,Slide4

RichardSocher

5/27/15

Composi8on:BigramandRecursivefunc8ons

?

ManyrelatedmodelsarespecialcasesofMV---RNN

?

MitchellandLapata,2010;Zanzo^oetal.,2010:

?

BaroniandZamparelli(2010):Aisanadjec8vematrixandbis

anounvector

?

RNNsofSocheretal.2011(ICML,EMNLP,NIPS)arealso

specialcases

?

Recursiveneuraltensornetworksbringquadra8cand

mul8plica8veinterac8onsbetweenvectors

Addi8onalchoiceforrecursiveneuralnets

?

Dependencytreesfocusmoreonseman8cstructure

1.

Cons8tuencyTree

2.

DependencyTree

3.

BalancedTree

Composi8on:CNNs

?

Severalvariantsalso:

?

?

?

Nopoolinglayers

Poolinglayers:simplemax---poolingordynamicpoolingPoolingacrossdi?erentdimensions

?

SomewhatlessexploredinNLPthanRNNs2

?

Notlinguis8callynorcogni8velyplausible

Lecture1,Slide7

RichardSocher

5/27/15

Composi8on:RecurrentNeuralNets

?

Vanilla

?

GRU

?

LSTM

?

ManyvariantsofLSTMs

“LSTM:ASearchSpaceOdyssey”byGre?etal.2015

Lecture1,Slide8

RichardSocher

5/27/15

Modeloverview:Objec8vefunc8on

?

Max---margin

?

Cross---entropy

Supervisedtopredictaclass

Unsupervised:predictsurroundingwords

?

Auto---encoder

Myopinion:UnclearbenefitsforNLP

Unlessencodinganothermodality

Lecture1,Slide9

RichardSocher

5/27/15

Op8miza8on

?

Ini8aliza8on(wordvectorandcomposi8onparameters)!!

?

Op8miza8onalgorithm

?

?

?

?

?

SGD

SGD+momentumL---BFGS

AdaGrad

Adelta

?

Op8miza8ontricks

Regulariza8on(somedefineaspartofmodel)

Dropout

Lecture1,Slide10

RichardSocher

5/27/15

CharacterRNNsontextandcode

h^p://karpathy.github.io/2015/05/21/rnn---e?ec8veness/

Lecture1,Slide11

RichardSocher

5/27/15

CharacterRNNsontextandcode

?

Haven’tyetproducedusefulresultsonrealdatasets

?

ShowsthatRNNscanmemorizesequencesandkeepmemory

(mostlyLSTMs)

?

Mostinteres8ngresultssimplytrainondatasetandsample

fromitaRerwards(firstshownbySutskeveretal.2011:Genera8ngTextwithRecurrentNeuralNetworks)

?

ResultsfromanLSTM(karpathy.github.io)

Lecture1,Slide12

RichardSocher

5/27/15

Shakespeare

PANDARUS:

Alas,IthinkheshallbecomeapproachedandthedayWhenli^lesrainwouldbea^ain'dintobeingneverfed,Andwhoisbutachainandsubjectsofhisdeath,

Ishouldnotsleep.

SecondSenator:

Theyareawaythismiseries,produceduponmysoul,Breakingandstronglyshouldbeburied,whenIperish

Theearthandthoughtsofmanystates.

DUKEVINCENTIO:

Well,yourwitisinthecareofsideandthat.

Lecture1,Slide13

RichardSocher

5/27/15

Wikipedia

NaturalismanddecisionforthemajorityofArabcountries'

capitalidewasgroundedbytheIrishlanguageby[[JohnClair]],[[AnImperialJapaneseRevolt]],associatedwithGuangzham'ssovereignty.HisgeneralswerethepowerfulrulerofthePortugalinthe[[ProtestantImmineners]],whichcouldbesaidtobedirectlyinCantoneseCommunica8on,whichfollowedaceremonyandsetinspiredprison,training.Theemperortravelledbackto[[An8och,Perth,October25|21]]tonote,theKingdomofCostaRica,unsuccessfulfashionedthe[[Thrales]],[[Cynth'sDajoard]],knowninwestern[[Scotland]],nearItalytotheconquestofIndiawiththeconflict.

Lecture1,Slide14

RichardSocher

5/27/15

Latex(hadtobefixedmanually)

Lecture1,Slide15

RichardSocher

5/27/15

Code!(Linuxsourcecode)

Lecture1,Slide16

RichardSocher

5/27/15

Morphology

?

Be^erWordRepresenta8onswithRecursiveNeuralNetworks

forMorphology–Luongetal.(slidesfromLuong)

?

Problemwithwordvectors:

poorlyes8materareandcomplexwords.

dis8nct

dis8nctnessa?ect

una?ected

di?erentdis8nc8vebroadernarrower

morphologiespesawatcleRspathologiesexacerbateimpactscharacterize

unno8ceddwarfedmi8gated

uniquebroaddis8nc8veseparate

companionroskamhitoshienjoyedallowpreventinvolveenable

mon8sheathskrystal

Lecture1,Slide17

RichardSocher

5/27/15

Limita8onsofexis8ngwork

?

?

Treatrelatedwordsasindependenten88es.

Representunknownwordswithafewvectors.

35323

23828

22687

352634791285

569

406

175 141

108

WordfrequenciesinWikipediadocs(986mtokens)

Luong’sapproach–networkstructure

vT

f(W[x;x;...;x]+b)

1 2

n

v

W,b

unfortunately

Wm,bm

the

bank

was

closed

Wm,bm

unfortunate

ly

close

d

Wm,bm

p=f(Wm[xstem;xaffix]+bm)

un

fortunate

?

NeuralLanguageModel:simplefeed---forwardnetwork(Huang,etal.,2012)

withranking---typecost(Collobertetal.,2011).

?

MorphologyModel:recursiveneuralnetwork(Socheretal.,2011).

MorphologyModel

NeuralLanguageModel

Analysis

?

Blendswordstructureandsyntac8c---seman8cinforma8on.

commen8ng

una?ecteddis8nctdis8nctnessheartlessness

saudi---owned

insis8nginsistedfocusing

unno8ceddwarfedmi8gateddi?erentdis8nc8vebroadermorphologiespesawatcleRs

?

avatarmohajirkripalani

commentedcommentscri8cizing

undesiredunhinderedunrestricteddivergentdiversedis8nc8vedis8nc8venesssmallnesslargenesscorrup8veinhumanityine?ectual

saudi---basedsyrian---controlled

Solu8onstotheproblemofpolysemouswords

?

ImprovingWordRepresenta8onsViaGlobalContextAnd

Mul8pleWordPrototypesbyHuangetal.2012

Lecture1,Slide21

RichardSocher

5/27/15

Naturallanguageinference

Claim:Simpletasktodefine,butengagesthefullcomplexityof

composi8onalseman8cs:

?

?

?

?

?

?

?

?

Lexicalentailment

Quan8fica8onCoreferenceLexical/scopeambiguity

Commonsenseknowledge

Proposi8onalawtudesModality

Fac8vityandimplica8vity

Lecture1,Slide22

RichardSocher

5/27/15

Firsttrainingdata

?

?

?

?

?

?

?

Trainingdata:

dance

waltztangosleep

waltz

entails

neutralentailscontradicts

entails

move

tangodance

dance

dance

Memoriza8on(trainingset): Generaliza8on(testset):

?

?

dance

waltz

???move

???tango

sleep???waltz

tango???move

Lecture1,Slide23

RichardSocher

5/27/15

Naturallanguageinference:defini8ons!

Lecture1,Slide24

RichardSocher

5/27/15

AminimalNNforlexicalrela8ons

?

Wordsarelearnedembeddingvectors.

?

OneplainRNNorRNTNlayer

?

SoRmaxemitsrela8onlabels

?

LearneverythingwithSGD.

Lecture1,Slide25

RichardSocher

5/27/15

Recursioninproposi8onallogic

Experimentalapproach:Trainonrela8onalstatementsgenerated

fromsomeformalsystem,testonothersuchrela8onalstatements.

Themodelneedsto:

?

?

Learntherela8onsbetweenindividualwords.(lexicalrela8ons)

Learnhowlexicalrela8onsimpactphrasalrela8ons.

Thisneedstoberecursivelyapplicable!

?

a≡a,

a^(nota),

a≡(not(nota)),

...

Lecture1,Slide26

RichardSocher

5/27/15

NaturallanguageinferencewithRNNs

?

Twotrees+learnedcomparisonlayer,thenaclassifier:

Lecture1,Slide27

RichardSocher

5/27/15

NaturallanguageinferencewithRNNs

Lecture1,Slide28

RichardSocher

5/27/15

Ques8onAnswering:QuizBowlCompe88on

?

QUESTION:

HeleRunfinishedanovelwhose8tlecharacterforgeshisfather'ssignaturetogetoutofschoolandavoidsthedraRbyfeigningdesiretojoin.AmorefamousworkbythisauthortellsoftheriseandfallofthecomposerAdrianLeverkühn.AnotherofhisnovelsfeaturesthejesuitNapthaandhisopponentSe^embrini,whilehismostfamousworkdepictstheagingwriterGustavvonAschenbach.NamethisGermanauthorofTheMagicMountainandDeathinVenice.

?

Iyyeretal.2014:ANeuralNetworkforFactoidQues8on

AnsweringoverParagraphs

Ques8onAnswering:QuizBowlCompe88on

?

QUESTION:

HeleRunfinishedanovelwhose8tlecharacterforgeshisfather'ssignaturetogetoutofschoolandavoidsthedraRbyfeigningdesiretojoin.AmorefamousworkbythisauthortellsoftheriseandfallofthecomposerAdrianLeverkühn.AnotherofhisnovelsfeaturesthejesuitNapthaandhisopponentSe^embrini,whilehismostfamousworkdepictstheagingwriterGustavvonAschenbach.NamethisGermanauthorofTheMagicMountainandDeathinVenice.

ANSWER:ThomasMann

?

RecursiveNeuralNetworks

?

Followdependencystructure

PushingFactsintoEn8tyVectors

QantaModelCanDefeatHumanPlayers

LiteratureQues8onsareHard!

VisualGrounding

?

Idea:Mapsentencesandimagesintoajointspace

?

Socheretal.2013:

GroundedComposi8onalSeman8csforFindingandDescribingImageswithSentences

Discussion:Composi8onalStructure

?

RecursiveNeuralNetworkssofar

usedcons8tuencytrees

whichresultsinmoresyntac8callyinfluencedrepresenta8ons

?

Instead:Usedependencytreeswhichcapturemore

seman8cstructure

Convolu8onalNeuralNetworkforImages

?

?

CNNtrainedonImageNe