《人工神經(jīng)網(wǎng)絡(luò)》PPT課件.ppt

人工神經(jīng)網(wǎng)絡(luò),中國科學院自動化研究所 吳高巍 2016-11-29,聯(lián)結(jié)主義學派,又稱仿生學派或生理學派 認為人的思維基元是神經(jīng)元，而不是符號處理過程 認為人腦不同于電腦 核心：智能的本質(zhì)是聯(lián)接機制。 原理：神經(jīng)網(wǎng)絡(luò)及神經(jīng)網(wǎng)絡(luò)間的連接機制和學習算法,麥卡洛可（McCulloch）,皮茨（Pitts),什么是神經(jīng)網(wǎng)絡(luò),所謂的人工神經(jīng)網(wǎng)絡(luò)就是基于模仿生物大腦的結(jié)構(gòu)和功能而構(gòu)成的一種信息處理系統(tǒng)（計算機）。 個體單元相互連接形成多種類型結(jié)構(gòu)的圖 循環(huán)、非循環(huán) 有向、無向 自底向上（Bottom-Up）AI 起源于生物神經(jīng)系統(tǒng) 從結(jié)構(gòu)模擬到功能模擬,仿生,人工神經(jīng)網(wǎng)絡(luò),內(nèi)容,生物學啟示 多層神經(jīng)網(wǎng)絡(luò) Hopfield網(wǎng)絡(luò) 自組織網(wǎng)絡(luò),生物學啟示,神經(jīng)元組成：細胞體，軸突，樹突，突觸 神經(jīng)元之間通過突觸兩兩相連。信息的傳遞發(fā)生在突觸。 突觸記錄了神經(jīng)元間聯(lián)系的強弱。 只有達到一定的興奮程度，神經(jīng)元才向外界傳輸信息。,生物神經(jīng)元,神經(jīng)元,神經(jīng)元特性 信息以預知的確定方向傳遞 一個神經(jīng)元的樹突細胞體軸突突觸另一個神經(jīng)元樹突 時空整合性 對不同時間通過同一突觸傳入的信息具有時間整合功能 對同一時間通過不同突觸傳入的信息具有空間整合功能,神經(jīng)元,工作狀態(tài) 興奮狀態(tài)，對輸入信息整合后使細胞膜電位升高，當高于動作電位的閾值時，產(chǎn)生神經(jīng)沖動，并由軸突輸出。 抑制狀態(tài)，對輸入信息整合后使細胞膜電位降低，當?shù)陀趧幼麟娢坏拈撝禃r，無神經(jīng)沖動產(chǎn)生。 結(jié)構(gòu)的可塑性 神經(jīng)元之間的柔性連接：突觸的信息傳遞特性是可變的 學習記憶的基礎(chǔ),神經(jīng)元模型,從生物學結(jié)構(gòu)到數(shù)學模型,人工神經(jīng)元,M-P模型,McClloch and Pitts, A logical calculus of the ideas immanent in nervous activity, 1943,f: 激活函數(shù)(Activation Function) g: 組合函數(shù)(Combination Function),Weighted Sum Radial Distance,組合函數(shù),(e) (f),Threshold,Linear,Saturating Linear,Logistic Sigmoid,Hyperbolic tangent Sigmoid,Gaussian,激活函數(shù),人工神經(jīng)網(wǎng)絡(luò),多個人工神經(jīng)元按照特定的網(wǎng)絡(luò)結(jié)構(gòu)聯(lián)接在一起，就構(gòu)成了一個人工神經(jīng)網(wǎng)絡(luò)。 神經(jīng)網(wǎng)絡(luò)的目標就是將輸入轉(zhuǎn)換成有意義的輸出。,生物系統(tǒng)中的學習,自適應(yīng)學習 適應(yīng)的目標是基于對環(huán)境信息的響應(yīng)獲得更好的狀態(tài) 在神經(jīng)層面上，通過突觸強度的改變實現(xiàn)學習 消除某些突觸，建立一些新的突觸,生物系統(tǒng)中的學習,Hebb學習律 神經(jīng)元同時激活，突觸強度增加 異步激活，突觸強度減弱 學習律符合能量最小原則 保持突觸強度需要能量，所以在需要的地方保持，在不需要的地方不保持。,ANN的學習規(guī)則,能量最小 ENERGY MINIMIZATION 對人工神經(jīng)網(wǎng)絡(luò)，需要確定合適的能量定義；可以使用數(shù)學上的優(yōu)化技術(shù)來發(fā)現(xiàn)如何改變神經(jīng)元間的聯(lián)接權(quán)重。,ENERGY = measure of task performance error,兩個主要問題 結(jié)構(gòu) How to interconnect individual units? 學習方法 How to automatically determine the connection weights or even structure of ANN?,Solutions to these two problems leads to a concrete ANN!,人工神經(jīng)網(wǎng)絡(luò),前饋結(jié)構(gòu)(Feedforward Architecture) - without loops - static 反饋/循環(huán)結(jié)構(gòu)(Feedback/Recurrent Architecture) - with loops - dynamic (non-linear dynamical systems),ANN結(jié)構(gòu),General structures of feedforward networks,General structures of feedback networks,通過神經(jīng)網(wǎng)絡(luò)所在環(huán)境的模擬過程，調(diào)整網(wǎng)絡(luò)中的自由參數(shù) Learning by data 學習模型 Incremental vs. Batch 兩種類型 Supervised vs. Unsupervised,ANN的學習方法,若兩端的神經(jīng)元同時激活，增強聯(lián)接權(quán)重 Unsupervised Learning,學習策略: Hebbrian Learning,最小化實際輸出與期望輸出之間的誤差(Supervised) - Delta Rule (LMS Rule, Widrow-Hoff) - B-P Learning,Objective:,Solution:,學習策略: Error Correction,采用隨機模式，跳出局部極小 - 如果網(wǎng)絡(luò)性能提高，新參數(shù)被接受. - 否則，新參數(shù)依概率接受,學習策略: Stochastic Learning,“勝者為王”(Winner-take-all ) Unsupervised How to compete? - Hard competition Only one neuron is activated - Soft competition Neurons neighboring the true winner are activated.,學習策略: Competitive Learning,重要的人工神經(jīng)網(wǎng)絡(luò)模型,多層神經(jīng)網(wǎng)絡(luò) 徑向基網(wǎng)絡(luò) Hopfield網(wǎng)絡(luò) Boltzmann機 自組織網(wǎng)絡(luò) ,多層感知機(MLP),感知機實質(zhì)上是一種神經(jīng)元模型 閾值激活函數(shù),Rosenblatt, 1957,感知機,判別規(guī)則,輸入空間中 樣本是空間中的一個點 權(quán)向量是一個超平面 超平面一邊對應(yīng) Y=1 另一邊對應(yīng) Y=-1,單層感知機學習,調(diào)整權(quán)值，減少訓練集上的誤差 簡單的權(quán)值更新規(guī)則: 初始化 對每一個訓練樣本: Classify with current weights If correct, no change! If wrong: adjust the weight vector,30,學習: Binary Perceptron,初始化 對每一個訓練樣本: Classify with current weights If correct (i.e., y=y*), no change! If wrong: adjust the weight vector by adding or subtracting the feature vector. Subtract if y* is -1.,多類判別情況,If we have multiple classes: A weight vector for each class: Score (activation) of a class y: Prediction highest score wins,學習: Multiclass Perceptron,初始化 依次處理每個樣本 Predict with current weights If correct, no change! If wrong: lower score of wrong answer, raise score of right answer,感知機特性,可分性: true if some parameters get the training set perfectly correct Can represent AND, OR, NOT, etc., but not XOR 收斂性: if the training is separable, perceptron will eventually converge (binary case),Separable,Non-Separable,感知機存在的問題,噪聲（不可分情況）: if the data isnt separable, weights might thrash 泛化性: finds a “barely” separating solution,改進感知機,線性可分情況,Which of these linear separators is optimal?,Support Vector Machines,Maximizing the margin: good according to intuition, theory, practice Only support vectors matter; other training examples are ignorable Support vector machines (SVMs) find the separator with max margin,SVM,優(yōu)化學習,問題描述 訓練數(shù)據(jù) 目標：發(fā)現(xiàn)最好的權(quán)值，使得對每一個樣本x的輸出都符合類別標簽 樣本xi的標簽可等價于標簽向量 采用不同的激活函數(shù) 平方損失：,單層感知機,單層感知機,單層感知機,單層感知機,采用線性激活函數(shù)，權(quán)值向量具有解析解 批處理模式 一次性更新權(quán)重 缺點：收斂慢 增量模式 逐樣本更新權(quán)值 隨機近似，但速度快并能保證收斂,多層感知機 (MLP),層間神經(jīng)元全連接,MLPs表達能力,3 layers: All continuous functions 4 layers: all functions,How to learn the weights?,waiting B-P algorithm until 1986,B-P Network,結(jié)構(gòu) A kind of multi-layer perceptron, in which the Sigmoid activation function is used.,B-P 算法,學習方法 - Input data was put forward from input layer to hidden layer, then to out layer - Error information was propagated backward from out layer to hidder layer, then to input layer,Rumelhart & Meclelland, Nature,1986,B-P 算法,Global Error Measure,desired output,generated output,squared error,The objective is to minimize the squared error, i.e. reach the Minimum Squared Error (MSE),B-P 算法,Step1. Select a pattern from the training set and present it to the network. Step2. Compute activation of input, hidden and output neurons in that sequence. Step3. Compute the error over the output neurons by comparing the generated outputs with the desired outputs. Step4. Use the calculated error to update all weights in the network, such that a global error measure gets reduced. Step5. Repeat Step1 through Step4 until the global error falls below a predefined threshold.,梯度下降方法,Optimization method for finding out the weight vector leading to the MSE,learning rate,gradient,vector form:,element form:,權(quán)值更新規(guī)則,For output layer:,權(quán)值更新規(guī)則,For output layer:,權(quán)值更新規(guī)則,For hidden layer,權(quán)值更新規(guī)則,For hidden layer,應(yīng)用: Handwritten digit recognition,3-nearest-neighbor = 2.4% error 40030010 unit MLP = 1.6% error LeNet: 768 192 30 10 unit MLP = 0.9% error Current best (SVMs) 0.4% error,MLPs：討論,實際應(yīng)用中 Preprocessing is important Normalize each dimension of data to -1, 1 Adapting the learning rate t = 1/t,MLPs：討論,優(yōu)點： 很強的表達能力 容易執(zhí)行 缺點： 收斂速度慢 過擬合（Over-fitting） 局部極小,采用Newton法,加正則化項，約束權(quán)值的平滑性 采用更少（但足夠數(shù)量）的隱層神經(jīng)元,嘗試不同的初始化 增加擾動,Hopfield 網(wǎng)絡(luò),反饋 結(jié)構(gòu) 可用加權(quán)無向圖表示 Dynamic System 兩種類型 Discrete (1982) and Continuous (science, 1984), by Hopfield,Hopfield網(wǎng)絡(luò),Combination function：Weighted Sum Activation function：Threshold,吸引子與穩(wěn)定性,How do we “program” the solutions of the problem into stable states (attractors) of the network? How do we ensure that the feedback system designed is stable? Lyapunovs modern stability theory allows us to investigate the stability problem by making use of a continuous scalar function of the state vector, called a Lyapunov (Energy) Function.,Hopfield網(wǎng)絡(luò)的能量函數(shù),With input Without input,Hopfield 模型,Hopfield證明了異步Hopfield網(wǎng)絡(luò)是穩(wěn)定的，其中權(quán)值定義為 Whatever be the initial state of the network, the energy decreases continuously with time until the system settles down into any local minimum of the energy surface.,Hopfield 網(wǎng)絡(luò): 聯(lián)想記憶,Hopfield網(wǎng)絡(luò)的一個主要應(yīng)用 基于與數(shù)據(jù)部分相似的輸入，可以回想起數(shù)據(jù)本身（attractor state） 也稱作內(nèi)容尋址記憶(content-addressable memory).,Stored Pattern,Memory Association,虞臺文, Feedback Networks and Associative Memories,Hopfield 網(wǎng)絡(luò): Associative Memories,Stored Pattern,Memory Association,虞臺文, Feedback Networks and Associative Memories,Hopfield網(wǎng)絡(luò)的一個主要應(yīng)用 基于與數(shù)據(jù)部分相似的輸入，可以回想起數(shù)據(jù)本身（attractor state） 也稱作內(nèi)容尋址記憶(content-addressable memory).,How to store patterns?,=?,How to store patterns?,=?,: Dimension of the stored pattern,權(quán)值確定: 外積(Outer Product),Vector form: Element form: Why? Satisfy the Hopfield model,An example of Hopfield memory,虞臺文, Feedback Networks and Associative Memories,Stable,E=4,E=0,E=4,Recall the first pattern (x1),Stable,E=4,E=0,E=4,Recall the second pattern (x2),Hopfield 網(wǎng)絡(luò): 組合優(yōu)化(Combinatorial Optimization),Hopfield網(wǎng)絡(luò)的另一個主要應(yīng)用 將優(yōu)化目標函數(shù)轉(zhuǎn)換成能量函數(shù)(energy function) 網(wǎng)絡(luò)的穩(wěn)定狀態(tài)是優(yōu)化問題的解,例: Solve Traveling Salesman Problem (TSP),Given n cities with distances dij, what is the shortest tour?,Illustration of TSP Graph,1,2,3,4,5,6,7,8,9,10,11,Hopfield Network for TSP,=?,Hopfield Network for TSP,=,City matrix,Constraint 1. Each row can have only one neuron “on”. 2. Each column can have only one neuron “on”. 3. For a n-city problem, n neurons will be on.,Hopfield Network for TSP,1,2,4,3,5,Time,City,The salesman reaches city 5 at time 3.,Weight determination for TSP: Design Energy Function,Constraint-1,Constraint-2,Constraint-3,能量函數(shù)轉(zhuǎn)換為2DHopfield網(wǎng)絡(luò)形式,Network is built!,Hopfield網(wǎng)絡(luò)迭代（TSP ）,The initial state generated randomly goes to the stable state (solution) with minimum energy,A 4-city example 阮曉剛， 神經(jīng)計算科學，2006,自組織特征映射 (SOFM),What is SOFM?,Neural Network with Unsupervised Learning Dimensionality reduction concomitant with preservation of topological information. Three principals - Self-reinforcing - Competition - Cooperation,Structure of SOFM,競爭(Competition),Finding the best matching weight vector for the present input. Criterion for determining the winning neuron: Maximum Inner Product Minimum Euclidean Distance,合作(Co