《人工神經(jīng)網(wǎng)絡(luò)》PPT課件.ppt

人工神經(jīng)網(wǎng)絡(luò),中國(guó)科學(xué)院自動(dòng)化研究所 吳高巍 2016-11-29,聯(lián)結(jié)主義學(xué)派,又稱(chēng)仿生學(xué)派或生理學(xué)派 認(rèn)為人的思維基元是神經(jīng)元，而不是符號(hào)處理過(guò)程 認(rèn)為人腦不同于電腦 核心：智能的本質(zhì)是聯(lián)接機(jī)制。 原理：神經(jīng)網(wǎng)絡(luò)及神經(jīng)網(wǎng)絡(luò)間的連接機(jī)制和學(xué)習(xí)算法,麥卡洛可（McCulloch）,皮茨（Pitts),什么是神經(jīng)網(wǎng)絡(luò),所謂的人工神經(jīng)網(wǎng)絡(luò)就是基于模仿生物大腦的結(jié)構(gòu)和功能而構(gòu)成的一種信息處理系統(tǒng)（計(jì)算機(jī)）。 個(gè)體單元相互連接形成多種類(lèi)型結(jié)構(gòu)的圖 循環(huán)、非循環(huán) 有向、無(wú)向 自底向上（Bottom-Up）AI 起源于生物神經(jīng)系統(tǒng) 從結(jié)構(gòu)模擬到功能模擬,仿生,人工神經(jīng)網(wǎng)絡(luò),內(nèi)容,生物學(xué)啟示 多層神經(jīng)網(wǎng)絡(luò) Hopfield網(wǎng)絡(luò) 自組織網(wǎng)絡(luò),生物學(xué)啟示,神經(jīng)元組成：細(xì)胞體，軸突，樹(shù)突，突觸 神經(jīng)元之間通過(guò)突觸兩兩相連。信息的傳遞發(fā)生在突觸。 突觸記錄了神經(jīng)元間聯(lián)系的強(qiáng)弱。 只有達(dá)到一定的興奮程度，神經(jīng)元才向外界傳輸信息。,生物神經(jīng)元,神經(jīng)元,神經(jīng)元特性 信息以預(yù)知的確定方向傳遞 一個(gè)神經(jīng)元的樹(shù)突細(xì)胞體軸突突觸另一個(gè)神經(jīng)元樹(shù)突 時(shí)空整合性 對(duì)不同時(shí)間通過(guò)同一突觸傳入的信息具有時(shí)間整合功能 對(duì)同一時(shí)間通過(guò)不同突觸傳入的信息具有空間整合功能,神經(jīng)元,工作狀態(tài) 興奮狀態(tài)，對(duì)輸入信息整合后使細(xì)胞膜電位升高，當(dāng)高于動(dòng)作電位的閾值時(shí)，產(chǎn)生神經(jīng)沖動(dòng)，并由軸突輸出。 抑制狀態(tài)，對(duì)輸入信息整合后使細(xì)胞膜電位降低，當(dāng)?shù)陀趧?dòng)作電位的閾值時(shí)，無(wú)神經(jīng)沖動(dòng)產(chǎn)生。 結(jié)構(gòu)的可塑性 神經(jīng)元之間的柔性連接：突觸的信息傳遞特性是可變的 學(xué)習(xí)記憶的基礎(chǔ),神經(jīng)元模型,從生物學(xué)結(jié)構(gòu)到數(shù)學(xué)模型,人工神經(jīng)元,M-P模型,McClloch and Pitts, A logical calculus of the ideas immanent in nervous activity, 1943,f: 激活函數(shù)(Activation Function) g: 組合函數(shù)(Combination Function),Weighted Sum Radial Distance,組合函數(shù),(e) (f),Threshold,Linear,Saturating Linear,Logistic Sigmoid,Hyperbolic tangent Sigmoid,Gaussian,激活函數(shù),人工神經(jīng)網(wǎng)絡(luò),多個(gè)人工神經(jīng)元按照特定的網(wǎng)絡(luò)結(jié)構(gòu)聯(lián)接在一起，就構(gòu)成了一個(gè)人工神經(jīng)網(wǎng)絡(luò)。 神經(jīng)網(wǎng)絡(luò)的目標(biāo)就是將輸入轉(zhuǎn)換成有意義的輸出。,生物系統(tǒng)中的學(xué)習(xí),自適應(yīng)學(xué)習(xí) 適應(yīng)的目標(biāo)是基于對(duì)環(huán)境信息的響應(yīng)獲得更好的狀態(tài) 在神經(jīng)層面上，通過(guò)突觸強(qiáng)度的改變實(shí)現(xiàn)學(xué)習(xí) 消除某些突觸，建立一些新的突觸,生物系統(tǒng)中的學(xué)習(xí),Hebb學(xué)習(xí)律 神經(jīng)元同時(shí)激活，突觸強(qiáng)度增加 異步激活，突觸強(qiáng)度減弱 學(xué)習(xí)律符合能量最小原則 保持突觸強(qiáng)度需要能量，所以在需要的地方保持，在不需要的地方不保持。,ANN的學(xué)習(xí)規(guī)則,能量最小 ENERGY MINIMIZATION 對(duì)人工神經(jīng)網(wǎng)絡(luò)，需要確定合適的能量定義；可以使用數(shù)學(xué)上的優(yōu)化技術(shù)來(lái)發(fā)現(xiàn)如何改變神經(jīng)元間的聯(lián)接權(quán)重。,ENERGY = measure of task performance error,兩個(gè)主要問(wèn)題 結(jié)構(gòu) How to interconnect individual units? 學(xué)習(xí)方法 How to automatically determine the connection weights or even structure of ANN?,Solutions to these two problems leads to a concrete ANN!,人工神經(jīng)網(wǎng)絡(luò),前饋結(jié)構(gòu)(Feedforward Architecture) - without loops - static 反饋/循環(huán)結(jié)構(gòu)(Feedback/Recurrent Architecture) - with loops - dynamic (non-linear dynamical systems),ANN結(jié)構(gòu),General structures of feedforward networks,General structures of feedback networks,通過(guò)神經(jīng)網(wǎng)絡(luò)所在環(huán)境的模擬過(guò)程，調(diào)整網(wǎng)絡(luò)中的自由參數(shù) Learning by data 學(xué)習(xí)模型 Incremental vs. Batch 兩種類(lèi)型 Supervised vs. Unsupervised,ANN的學(xué)習(xí)方法,若兩端的神經(jīng)元同時(shí)激活，增強(qiáng)聯(lián)接權(quán)重 Unsupervised Learning,學(xué)習(xí)策略: Hebbrian Learning,最小化實(shí)際輸出與期望輸出之間的誤差(Supervised) - Delta Rule (LMS Rule, Widrow-Hoff) - B-P Learning,Objective:,Solution:,學(xué)習(xí)策略: Error Correction,采用隨機(jī)模式，跳出局部極小 - 如果網(wǎng)絡(luò)性能提高，新參數(shù)被接受. - 否則，新參數(shù)依概率接受,學(xué)習(xí)策略: Stochastic Learning,“勝者為王”(Winner-take-all ) Unsupervised How to compete? - Hard competition Only one neuron is activated - Soft competition Neurons neighboring the true winner are activated.,學(xué)習(xí)策略: Competitive Learning,重要的人工神經(jīng)網(wǎng)絡(luò)模型,多層神經(jīng)網(wǎng)絡(luò) 徑向基網(wǎng)絡(luò) Hopfield網(wǎng)絡(luò) Boltzmann機(jī) 自組織網(wǎng)絡(luò) ,多層感知機(jī)(MLP),感知機(jī)實(shí)質(zhì)上是一種神經(jīng)元模型 閾值激活函數(shù),Rosenblatt, 1957,感知機(jī),判別規(guī)則,輸入空間中 樣本是空間中的一個(gè)點(diǎn) 權(quán)向量是一個(gè)超平面 超平面一邊對(duì)應(yīng) Y=1 另一邊對(duì)應(yīng) Y=-1,單層感知機(jī)學(xué)習(xí),調(diào)整權(quán)值，減少訓(xùn)練集上的誤差 簡(jiǎn)單的權(quán)值更新規(guī)則: 初始化 對(duì)每一個(gè)訓(xùn)練樣本: Classify with current weights If correct, no change! If wrong: adjust the weight vector,30,學(xué)習(xí): Binary Perceptron,初始化 對(duì)每一個(gè)訓(xùn)練樣本: Classify with current weights If correct (i.e., y=y*), no change! If wrong: adjust the weight vector by adding or subtracting the feature vector. Subtract if y* is -1.,多類(lèi)判別情況,If we have multiple classes: A weight vector for each class: Score (activation) of a class y: Prediction highest score wins,學(xué)習(xí): Multiclass Perceptron,初始化 依次處理每個(gè)樣本 Predict with current weights If correct, no change! If wrong: lower score of wrong answer, raise score of right answer,感知機(jī)特性,可分性: true if some parameters get the training set perfectly correct Can represent AND, OR, NOT, etc., but not XOR 收斂性: if the training is separable, perceptron will eventually converge (binary case),Separable,Non-Separable,感知機(jī)存在的問(wèn)題,噪聲（不可分情況）: if the data isnt separable, weights might thrash 泛化性: finds a “barely” separating solution,改進(jìn)感知機(jī),線(xiàn)性可分情況,Which of these linear separators is optimal?,Support Vector Machines,Maximizing the margin: good according to intuition, theory, practice Only support vectors matter; other training examples are ignorable Support vector machines (SVMs) find the separator with max margin,SVM,優(yōu)化學(xué)習(xí),問(wèn)題描述 訓(xùn)練數(shù)據(jù) 目標(biāo)：發(fā)現(xiàn)最好的權(quán)值，使得對(duì)每一個(gè)樣本x的輸出都符合類(lèi)別標(biāo)簽 樣本xi的標(biāo)簽可等價(jià)于標(biāo)簽向量 采用不同的激活函數(shù) 平方損失：,單層感知機(jī),單層感知機(jī),單層感知機(jī),單層感知機(jī),采用線(xiàn)性激活函數(shù)，權(quán)值向量具有解析解 批處理模式 一次性更新權(quán)重 缺點(diǎn)：收斂慢 增量模式 逐樣本更新權(quán)值 隨機(jī)近似，但速度快并能保證收斂,多層感知機(jī) (MLP),層間神經(jīng)元全連接,MLPs表達(dá)能力,3 layers: All continuous functions 4 layers: all functions,How to learn the weights?,waiting B-P algorithm until 1986,B-P Network,結(jié)構(gòu) A kind of multi-layer perceptron, in which the Sigmoid activation function is used.,B-P 算法,學(xué)習(xí)方法 - Input data was put forward from input layer to hidden layer, then to out layer - Error information was propagated backward from out layer to hidder layer, then to input layer,Rumelhart & Meclelland, Nature,1986,B-P 算法,Global Error Measure,desired output,generated output,squared error,The objective is to minimize the squared error, i.e. reach the Minimum Squared Error (MSE),B-P 算法,Step1. Select a pattern from the training set and present it to the network. Step2. Compute activation of input, hidden and output neurons in that sequence. Step3. Compute the error over the output neurons by comparing the generated outputs with the desired outputs. Step4. Use the calculated error to update all weights in the network, such that a global error measure gets reduced. Step5. Repeat Step1 through Step4 until the global error falls below a predefined threshold.,梯度下降方法,Optimization method for finding out the weight vector leading to the MSE,learning rate,gradient,vector form:,element form:,權(quán)值更新規(guī)則,For output layer:,權(quán)值更新規(guī)則,For output layer:,權(quán)值更新規(guī)則,For hidden layer,權(quán)值更新規(guī)則,For hidden layer,應(yīng)用: Handwritten digit recognition,3-nearest-neighbor = 2.4% error 40030010 unit MLP = 1.6% error LeNet: 768 192 30 10 unit MLP = 0.9% error Current best (SVMs) 0.4% error,MLPs：討論,實(shí)際應(yīng)用中 Preprocessing is important Normalize each dimension of data to -1, 1 Adapting the learning rate t = 1/t,MLPs：討論,優(yōu)點(diǎn)： 很強(qiáng)的表達(dá)能力 容易執(zhí)行 缺點(diǎn)： 收斂速度慢 過(guò)擬合（Over-fitting） 局部極小,采用Newton法,加正則化項(xiàng)，約束權(quán)值的平滑性 采用更少（但足夠數(shù)量）的隱層神經(jīng)元,嘗試不同的初始化 增加擾動(dòng),Hopfield 網(wǎng)絡(luò),反饋 結(jié)構(gòu) 可用加權(quán)無(wú)向圖表示 Dynamic System 兩種類(lèi)型 Discrete (1982) and Continuous (science, 1984), by Hopfield,Hopfield網(wǎng)絡(luò),Combination function：Weighted Sum Activation function：Threshold,吸引子與穩(wěn)定性,How do we “program” the solutions of the problem into stable states (attractors) of the network? How do we ensure that the feedback system designed is stable? Lyapunovs modern stability theory allows us to investigate the stability problem by making use of a continuous scalar function of the state vector, called a Lyapunov (Energy) Function.,Hopfield網(wǎng)絡(luò)的能量函數(shù),With input Without input,Hopfield 模型,Hopfield證明了異步Hopfield網(wǎng)絡(luò)是穩(wěn)定的，其中權(quán)值定義為 Whatever be the initial state of the network, the energy decreases continuously with time until the system settles down into any local minimum of the energy surface.,Hopfield 網(wǎng)絡(luò): 聯(lián)想記憶,Hopfield網(wǎng)絡(luò)的一個(gè)主要應(yīng)用 基于與數(shù)據(jù)部分相似的輸入，可以回想起數(shù)據(jù)本身（attractor state） 也稱(chēng)作內(nèi)容尋址記憶(content-addressable memory).,Stored Pattern,Memory Association,虞臺(tái)文, Feedback Networks and Associative Memories,Hopfield 網(wǎng)絡(luò): Associative Memories,Stored Pattern,Memory Association,虞臺(tái)文, Feedback Networks and Associative Memories,Hopfield網(wǎng)絡(luò)的一個(gè)主要應(yīng)用 基于與數(shù)據(jù)部分相似的輸入，可以回想起數(shù)據(jù)本身（attractor state） 也稱(chēng)作內(nèi)容尋址記憶(content-addressable memory).,How to store patterns?,=?,How to store patterns?,=?,: Dimension of the stored pattern,權(quán)值確定: 外積(Outer Product),Vector form: Element form: Why? Satisfy the Hopfield model,An example of Hopfield memory,虞臺(tái)文, Feedback Networks and Associative Memories,Stable,E=4,E=0,E=4,Recall the first pattern (x1),Stable,E=4,E=0,E=4,Recall the second pattern (x2),Hopfield 網(wǎng)絡(luò): 組合優(yōu)化(Combinatorial Optimization),Hopfield網(wǎng)絡(luò)的另一個(gè)主要應(yīng)用 將優(yōu)化目標(biāo)函數(shù)轉(zhuǎn)換成能量函數(shù)(energy function) 網(wǎng)絡(luò)的穩(wěn)定狀態(tài)是優(yōu)化問(wèn)題的解,例: Solve Traveling Salesman Problem (TSP),Given n cities with distances dij, what is the shortest tour?,Illustration of TSP Graph,1,2,3,4,5,6,7,8,9,10,11,Hopfield Network for TSP,=?,Hopfield Network for TSP,=,City matrix,Constraint 1. Each row can have only one neuron “on”. 2. Each column can have only one neuron “on”. 3. For a n-city problem, n neurons will be on.,Hopfield Network for TSP,1,2,4,3,5,Time,City,The salesman reaches city 5 at time 3.,Weight determination for TSP: Design Energy Function,Constraint-1,Constraint-2,Constraint-3,能量函數(shù)轉(zhuǎn)換為2DHopfield網(wǎng)絡(luò)形式,Network is built!,Hopfield網(wǎng)絡(luò)迭代（TSP ）,The initial state generated randomly goes to the stable state (solution) with minimum energy,A 4-city example 阮曉剛， 神經(jīng)計(jì)算科學(xué)，2006,自組織特征映射 (SOFM),What is SOFM?,Neural Network with Unsupervised Learning Dimensionality reduction concomitant with preservation of topological information. Three principals - Self-reinforcing - Competition - Cooperation,Structure of SOFM,競(jìng)爭(zhēng)(Competition),Finding the best matching weight vector for the present input. Criterion for determining the winning neuron: Maximum Inner Product Minimum Euclidean Distance,合作(Co