內(nèi)容參考講稿recurrent neural

1、Machine Perception and Interaction Group (MPIG) .cn 跟我學(xué)CS231(7)袁洪慧MPIG Open Seminar 0220公眾號(hào):mpig_robotRecurrent Neural NetworkRNN的應(yīng)用RNN的分類(lèi)Recurrent Neural NetworkRNN的正向傳播Truncated Backpropagation LSTMOther RNN Variantssummary目錄RNN的應(yīng)用對(duì)于序列化的特征任務(wù)，都適合用RNN來(lái)解決：情感分析關(guān)鍵字提取語(yǔ)音識(shí)別機(jī)器翻譯股票分析RNN的分類(lèi)“Vanilla” Neural N

2、etworkVanilla Neural NetworksRecurrent Neural Networks: Process Sequencese.g. Image Captioning image - sequence of wordse.g. Sentiment Classification sequence of words - sentimente.g. Machine Translation seq of words - seq of wordse.g. Video classification on frame levelRecurrent Neural Networkusual

3、ly want to predict a vector at some time stepsRecurrent Neural NetworkhRecurrent Neural NetworkWe can process a sequence of vectors x by applying a recurrence formula at every time step: new statesome function with parameters Wold state input vector atsome time stepRecurrent Neural NetworkWe can pro

4、cess a sequence of vectors x by applying a recurrence formula at every time step:Notice: the same function and the same set of parameters are used at every time step.(Simple) Recurrent Neural NetworkThe state consists of a single “hidden” vector h:RNN的展開(kāi)圖RNN的正向傳播RNN: Computational GraphRe-use the sa

5、me weight matrix at every time-step：RNN: Computational Graph: Many to Many RNN: Computational Graph: Many to OneRNN: Computational Graph: One to ManySequence to Sequence: Many-to-one + one-to-manyMany to one: Encode input sequence in a single vectorOne to many: Produce output sequence from single in

6、put vectorTruncated Backpropagation Backpropagation through time梯度截?cái)啵℅radient Clipping）為梯度設(shè)置閾值，超過(guò)該閾值的梯度值都會(huì)被cut,這樣更新的幅度就不會(huì)過(guò)大，因此容易收斂。具體做法：Truncated Backpropagation through timeTruncated Backpropagation through timeVanilla RNN Gradient FlowComputing gradient of h0 involves many factors of W (and repeat

7、ed tanh) Bengio et al, “Learning long-term dependencies with gradient descent is difficult”, IEEE Transactions on Neural Networks, 1994 Pascanu et al, “On the difficulty of training recurrent neural networks”, ICML 2013Largest singular value 1: Exploding gradients Largest singular value 1: Vanishing

8、 gradients Gradient clipping: Scale Computing gradient gradient if its norm is too bigSimple-RNN在實(shí)際應(yīng)用中并不多，原因：如果輸入越長(zhǎng)的話，展開(kāi)的網(wǎng)絡(luò)就越深，對(duì)于“深度”網(wǎng)絡(luò)訓(xùn)練的困難最常見(jiàn)的是 Gradient Explode 和 Gradient Vanish 的問(wèn)題。Simple-RNN基于先前的詞預(yù)測(cè)下一個(gè)詞，但在一些更加復(fù)雜的場(chǎng)景中，例如，“I grew up in France I speak fluent French” “France”則需要更長(zhǎng)時(shí)間的預(yù)測(cè)，而隨著上下文之間的間隔不斷

9、增大時(shí)，Simple-RNN會(huì)喪失學(xué)習(xí)到連接如此遠(yuǎn)的信息的能力。LSTM(Long Short-Term Memory)Long Short Term Memory (LSTM)RNN和LSTM框圖 LSTM的核心思想逐步理解 LSTM之遺忘門(mén)逐步理解 LSTM之輸入門(mén)LSTM還需要記住東西，所以有了圖示“記憶門(mén)”。逐步理解 LSTM逐步理解 LSTM之輸出門(mén)Other RNN VariantsGRU(Gated Recurrent Unit)GRU是和LSTM功能幾乎一樣的另一種網(wǎng)絡(luò)。最終的模型比標(biāo)準(zhǔn)的 LSTM 模型要簡(jiǎn)單，也是非常流行的變體SummaryRNNs allow a lot

10、of flexibility in architecture design Vanilla RNNs are simple but dont work very well Common to use LSTM or GRU: their additive interactions improve gradient flow Backward flow of gradients in RNN can explode or vanish. Exploding is controlled with gradient clipping. Vanishing is controlled with additive interactions (LSTM) Better/simpler architectures are a hot