CNN卷積神經(jīng)網(wǎng)絡原理

./一、CNN卷積神經(jīng)網(wǎng)絡原理簡介本文主要是詳細地解讀CNN的實現(xiàn)代碼。如果你沒學習過CNN,在此推薦周曉藝師兄的博文：DeepLearning〔深度學習學習筆記整理系列之〔七,以及UFLDL上的卷積特征提取、池化CNN的最大特點就是稀疏連接〔局部感受和權(quán)值共享,如下面兩圖所示,左為稀疏連接,右為權(quán)值共享。稀疏連接和權(quán)值共享可以減少所要訓練的參數(shù),減少計算復雜度。至于CNN的結(jié)構(gòu),以經(jīng)典的LeNet5來說明：這個圖真是無處不在,一談CNN,必說LeNet5,這圖來自于這篇論文：Gradient-BasedLearningAppliedtoDocumentRecognition,論文很長,第7頁那里開始講LeNet5這個結(jié)構(gòu),建議看看那部分。我這里簡單說一下,LeNet5這張圖從左到右,先是input,這是輸入層,即輸入的圖片。input-layer到C1這部分就是一個卷積層〔convolution運算,C1到S2是一個子采樣層〔pooling運算,關(guān)于卷積和子采樣的具體過程可以參考下圖：然后,S2到C3又是卷積,C3到S4又是子采樣,可以發(fā)現(xiàn),卷積和子采樣都是成對出現(xiàn)的,卷積后面一般跟著子采樣。S4到C5之間是全連接的,這就相當于一個MLP的隱含層了〔如果你不清楚MLP,參考《DeepLearningtutorial〔3MLP多層感知機原理簡介+代碼詳解》。C5到F6同樣是全連接,也是相當于一個MLP的隱含層。最后從F6到輸出output,其實就是一個分類器,這一層就叫分類層。ok,CNN的基本結(jié)構(gòu)大概就是這樣,由輸入、卷積層、子采樣層、全連接層、分類層、輸出這些基本"構(gòu)件"組成,一般根據(jù)具體的應用或者問題,去確定要多少卷積層和子采樣層、采用什么分類器。當確定好了結(jié)構(gòu)以后,如何求解層與層之間的連接參數(shù)？一般采用向前傳播〔FP+向后傳播〔BP的方法來訓練。具體可參考上面給出的鏈接。二、CNN卷積神經(jīng)網(wǎng)絡代碼詳細解讀〔基于python+theano代碼來自于深度學習教程：ConvolutionalNeuralNetworks<LeNet>,這個代碼實現(xiàn)的是一個簡化了的LeNet5,具體如下：沒有實現(xiàn)location-specificgainandbiasparameters用的是maxpooling,而不是average_pooling分類器用的是softmax,LeNet5用的是rbfLeNet5第二層并不是全連接的,本程序?qū)崿F(xiàn)的是全連接另外,代碼里將卷積層和子采用層合在一起,定義為"LeNetConvPoolLayer"〔卷積采樣層,這好理解,因為它們總是成對出現(xiàn)。但是有個地方需要注意,代碼中將卷積后的輸出直接作為子采樣層的輸入,而沒有加偏置b再通過sigmoid函數(shù)進行映射,即沒有了下圖中fx后面的bx以及sigmoid映射,也即直接由fx得到Cx。最后,代碼中第一個卷積層用的卷積核有20個,第二個卷積層用50個,而不是上面那張LeNet5圖中所示的6個和16個。了解了這些,下面看代碼：〔1導入必要的模塊[python]viewplaincopyimport

cPickle

import

gzip

import

os

import

sys

import

time

import

numpy

import

theano

import

theano.tensor

as

T

from

theano.tensor.signal

import

downsample

from

theano.tensor.nnet

import

conv

〔2定義CNN的基本"構(gòu)件"CNN的基本構(gòu)件包括卷積采樣層、隱含層、分類器,如下定義LeNetConvPoolLayer〔卷積+采樣層見代碼注釋：[python]viewplaincopy"""卷積+下采樣合成一個層LeNetConvPoolLayerrng:隨機數(shù)生成器,用于初始化Winput:4維的向量,filter_shape:<number

of

filters,

num

input

feature

maps,filter

height,

filter

width>image_shape:<batch

size,

num

input

feature

maps,image

height,

image

width>poolsize:

<#rows,

#cols>"""class

LeNetConvPoolLayer<object>:

def

__init__<self,

rng,

input,

filter_shape,

image_shape,

poolsize=<2,

2>>:

#assert

condition,condition為True,則繼續(xù)往下執(zhí)行,condition為False,中斷程序#image_shape[1]和filter_shape[1]都是num

input

feature

maps,它們必須是一樣的。assert

image_shape[1]

==

filter_shape[1]

self.input

=

input

#每個隱層神經(jīng)元〔即像素與上一層的連接數(shù)為num

input

feature

maps

*

filter

height

*

filter

width。#可以用d<filter_shape[1:]>來求得

fan_in

=

d<filter_shape[1:]>

#lower

layer上每個神經(jīng)元獲得的梯度來自于："num

output

feature

maps

*

filter

height

*

filter

width"

/pooling

size

fan_out

=

<filter_shape[0]

*

d<filter_shape[2:]>

/

d<poolsize>>

#以上求得fan_in、fan_out

,將它們代入公式,以此來隨機初始化W,W就是線性卷積核

W_bound

=

numpy.sqrt<6.

/

<fan_in

+

fan_out>>

self.W

=

theano.shared<

numpy.asarray<

rng.uniform<low=-W_bound,

high=W_bound,

size=filter_shape>,

dtype=theano.config.floatX

>,

borrow=True

>

#

the

bias

is

a

1D

tensor

--

one

bias

per

output

feature

map#偏置b是一維向量,每個輸出圖的特征圖都對應一個偏置,#而輸出的特征圖的個數(shù)由filter個數(shù)決定,因此用filter_shape[0]即number

of

filters來初始化

b_values

=

numpy.zeros<<filter_shape[0],>,

dtype=theano.config.floatX>

self.b

=

theano.shared<value=b_values,

borrow=True>

#將輸入圖像與filter卷積,conv.conv2d函數(shù)#卷積完沒有加b再通過sigmoid,這里是一處簡化。

conv_out

=

conv.conv2d<

input=input,

filters=self.W,

filter_shape=filter_shape,

image_shape=image_shape

>

#maxpooling,最大子采樣過程

pooled_out

=

downsample.max_pool_2d<

input=conv_out,

ds=poolsize,

ignore_border=True

>

#加偏置,再通過tanh映射,得到卷積+子采樣層的最終輸出#因為b是一維向量,這里用維度轉(zhuǎn)換函數(shù)dimshuffle將其reshape。比如b是<10,>,#則b.dimshuffle<'x',

0,

'x',

'x'>>將其reshape為<1,10,1,1>self.output

=

T.tanh<pooled_out

+

self.b.dimshuffle<'x',

0,

'x',

'x'>>

#卷積+采樣層的參數(shù)self.params

=

[self.W,

self.b]

定義隱含層HiddenLayer這個跟上一篇文章《DeepLearningtutorial〔3MLP多層感知機原理簡介+代碼詳解》中的HiddenLayer是一致的,直接拿過來：[python]viewplaincopy"""注釋：這是定義隱藏層的類,首先明確：隱藏層的輸入即input,輸出即隱藏層的神經(jīng)元個數(shù)。輸入層與隱藏層是全連接的。假設輸入是n_in維的向量〔也可以說時n_in個神經(jīng)元,隱藏層有n_out個神經(jīng)元,則因為是全連接,一共有n_in*n_out個權(quán)重,故W大小時<n_in,n_out>,n_in行n_out列,每一列對應隱藏層的每一個神經(jīng)元的連接權(quán)重。b是偏置,隱藏層有n_out個神經(jīng)元,故b時n_out維向量。rng即隨機數(shù)生成器,,用于初始化W。input訓練模型所用到的所有輸入,并不是MLP的輸入層,MLP的輸入層的神經(jīng)元個數(shù)時n_in,而這里的參數(shù)input大小是〔n_example,n_in,每一行一個樣本,即每一行作為MLP的輸入層。activation:激活函數(shù),這里定義為函數(shù)tanh"""class

HiddenLayer<object>:

def

__init__<self,

rng,

input,

n_in,

n_out,

W=None,

b=None,

activation=T.tanh>:

self.input

=

input

#類HiddenLayer的input即所傳遞進來的input"""

注釋：

代碼要兼容GPU,則必須使用

dtype=theano.config.floatX,并且定義為theano.shared

另外,W的初始化有個規(guī)則：如果使用tanh函數(shù),則在-sqrt<6./<n_in+n_hidden>>到sqrt<6./<n_in+n_hidden>>之間均勻

抽取數(shù)值來初始化W,若時sigmoid函數(shù),則以上再乘4倍。

"""#如果W未初始化,則根據(jù)上述方法初始化。#加入這個判斷的原因是：有時候我們可以用訓練好的參數(shù)來初始化W,見我的上一篇文章。if

W

isNone:

W_values

=

numpy.asarray<

rng.uniform<

low=-numpy.sqrt<6.

/

<n_in

+

n_out>>,

high=numpy.sqrt<6.

/

<n_in

+

n_out>>,

size=<n_in,

n_out>

>,

dtype=theano.config.floatX

>

if

activation

==

theano.tensor.nnet.sigmoid:

W_values

*=

4

W

=

theano.shared<value=W_values,

name='W',

borrow=True>

if

b

isNone:

b_values

=

numpy.zeros<<n_out,>,

dtype=theano.config.floatX>

b

=

theano.shared<value=b_values,

name='b',

borrow=True>

#用上面定義的W、b來初始化類HiddenLayer的W、bself.W

=

W

self.b

=

b

#隱含層的輸出

lin_output

=

T.dot<input,

self.W>

+

self.b

self.output

=

<

lin_output

if

activation

isNoneelse

activation<lin_output>

>

#隱含層的參數(shù)self.params

=

[self.W,

self.b]

定義分類器〔Softmax回歸采用Softmax,這跟《DeepLearningtutorial〔1Softmax回歸原理簡介+代碼詳解》中的LogisticRegression是一樣的,直接拿過來：[python]viewplaincopy"""定義分類層LogisticRegression,也即Softmax回歸在deeplearning

tutorial中,直接將LogisticRegression視為Softmax,而我們所認識的二類別的邏輯回歸就是當n_out=2時的LogisticRegression"""#參數(shù)說明：#input,大小就是<n_example,n_in>,其中n_example是一個batch的大小,#因為我們訓練時用的是Minibatch

SGD,因此input這樣定義#n_in,即上一層<隱含層>的輸出#n_out,輸出的類別數(shù)

class

LogisticRegression<object>:

def

__init__<self,

input,

n_in,

n_out>:

#W大小是n_in行n_out列,b為n_out維向量。即：每個輸出對應W的一列以及b的一個元素。

self.W

=

theano.shared<

value=numpy.zeros<

<n_in,

n_out>,

dtype=theano.config.floatX

>,

name='W',

borrow=True

>

self.b

=

theano.shared<

value=numpy.zeros<

<n_out,>,

dtype=theano.config.floatX

>,

name='b',

borrow=True

>

#input是<n_example,n_in>,W是〔n_in,n_out,點乘得到<n_example,n_out>,加上偏置b,#再作為的輸入,得到p_y_given_x#故p_y_given_x每一行代表每一個樣本被估計為各類別的概率

#PS：b是n_out維向量,與<n_example,n_out>矩陣相加,內(nèi)部其實是先復制n_example個b,#然后<n_example,n_out>矩陣的每一行都加bself.p_y_given_x

=

T.nnet.softmax<T.dot<input,

self.W>

+

self.b>

#argmax返回最大值下標,因為本例數(shù)據(jù)集是MNIST,下標剛好就是類別。axis=1表示按行操作。self.y_pred

=

T.argmax<self.p_y_given_x,

axis=1>

#params,LogisticRegression的參數(shù)

self.params

=

[self.W,

self.b]

到這里,CNN的基本"構(gòu)件"都有了,下面要用這些"構(gòu)件"組裝成LeNet5〔當然,是簡化的,上面已經(jīng)說了,具體來說,就是組裝成：LeNet5=input+LeNetConvPoolLayer_1+LeNetConvPoolLayer_2+HiddenLayer+LogisticRegression+output。然后將其應用于MNIST數(shù)據(jù)集,用BP算法去解這個模型,得到最優(yōu)的參數(shù)。〔3加載MNIST數(shù)據(jù)集〔[python]viewplaincopy"""加載MNIST數(shù)據(jù)集load_data<>"""def

load_data<dataset>:

#

dataset是數(shù)據(jù)集的路徑,程序首先檢測該路徑下有沒有MNIST數(shù)據(jù)集,沒有的話就下載MNIST數(shù)據(jù)集#這一部分就不解釋了,與softmax回歸算法無關(guān)。

data_dir,

data_file

=

os.path.split<dataset>

if

data_dir

==

""

andnot

os.path.isfile<dataset>:

#

Check

if

dataset

is

in

the

data

directory.

new_path

=

os.path.join<

os.path.split<__file__>[0],

"..",

"data",

dataset

>

if

os.path.isfile<new_path>

or

data_file

==

'mnist.pkl.gz':

dataset

=

new_path

if

<not

os.path.isfile<dataset>>

and

data_file

==

'mnist.pkl.gz':

import

urllib

origin

=

<

'http://www.iro.umontreal.ca/~lisa/deep/data/mnist/mnist.pkl.gz'

>

print'Downloading

data

from

%s'

%

origin

urllib.urlretrieve<origin,

dataset>

print'...

loading

data'#以上是檢測并下載數(shù)據(jù)集,不是本文重點。下面才是load_data的開始#從"mnist.pkl.gz"里加載train_set,

valid_set,

test_set,它們都是包括label的#主要用到python里的gzip.open<>函數(shù),以及

cPickle.load<>。#‘rb’表示以二進制可讀的方式打開文件

f

=

gzip.open<dataset,

'rb'>

train_set,

valid_set,

test_set

=

cPickle.load<f>

f.close<>

#將數(shù)據(jù)設置成shared

variables,主要時為了GPU加速,只有shared

variables才能存到GPU

memory中#GPU里數(shù)據(jù)類型只能是float。而data_y是類別,所以最后又轉(zhuǎn)換為int返回def

shared_dataset<data_xy,

borrow=True>:

data_x,

data_y

=

data_xy

shared_x

=

theano.shared<numpy.asarray<data_x,

dtype=theano.config.floatX>,

borrow=borrow>

shared_y

=

theano.shared<numpy.asarray<data_y,

dtype=theano.config.floatX>,

borrow=borrow>

return

shared_x,

T.cast<shared_y,

'int32'>

test_set_x,

test_set_y

=

shared_dataset<test_set>

valid_set_x,

valid_set_y

=

shared_dataset<valid_set>

train_set_x,

train_set_y

=

shared_dataset<train_set>

rval

=

[<train_set_x,

train_set_y>,

<valid_set_x,

valid_set_y>,

<test_set_x,

test_set_y>]

return

rval

〔4實現(xiàn)LeNet5并測試[python]viewplaincopy"""實現(xiàn)LeNet5LeNet5有兩個卷積層,第一個卷積層有20個卷積核,第二個卷積層有50個卷積核"""def

evaluate_lenet5<learning_rate=0.1,

n_epochs=200,

dataset='mnist.pkl.gz',

nkerns=[20,

50],

batch_size=500>:

"""

learning_rate:學習速率,隨機梯度前的系數(shù)。

n_epochs訓練步數(shù),每一步都會遍歷所有batch,即所有樣本

batch_size,這里設置為500,即每遍歷完500個樣本,才計算梯度并更新參數(shù)

nkerns=[20,

50],每一個LeNetConvPoolLayer卷積核的個數(shù),第一個LeNetConvPoolLayer有

20個卷積核,第二個有50個

"""

rng

=

numpy.random.RandomState<23455>

#加載數(shù)據(jù)

datasets

=

load_data<dataset>

train_set_x,

train_set_y

=

datasets[0]

valid_set_x,

valid_set_y

=

datasets[1]

test_set_x,

test_set_y

=

datasets[2]

#

計算batch的個數(shù)

n_train_batches

=

train_set_x.get_value<borrow=True>.shape[0]

n_valid_batches

=

valid_set_x.get_value<borrow=True>.shape[0]

n_test_batches

=

test_set_x.get_value<borrow=True>.shape[0]

n_train_batches

/=

batch_size

n_valid_batches

/=

batch_size

n_test_batches

/=

batch_size

#定義幾個變量,index表示batch下標,x表示輸入的訓練數(shù)據(jù),y對應其標簽

index

=

T.lscalar<>

x

=

T.matrix<'x'>

y

=

T.ivector<'y'>

#######################

BUILD

ACTUAL

MODEL

#######################print'...

building

the

model'#我們加載進來的batch大小的數(shù)據(jù)是<batch_size,

28

*

28>,但是LeNetConvPoolLayer的輸入是四維的,所以要reshape

layer0_input

=

x.reshape<<batch_size,

1,

28,

28>>

#

layer0即第一個LeNetConvPoolLayer層#輸入的單張圖片<28,28>,經(jīng)過conv得到<28-5+1

,

28-5+1>

=

<24,

24>,#經(jīng)過maxpooling得到<24/2,

24/2>

=

<12,

12>#因為每個batch有batch_size張圖,第一個LeNetConvPoolLayer層有nkerns[0]個卷積核,#故layer0輸出為<batch_size,

nkerns[0],

12,

12>

layer0

=

LeNetConvPoolLayer<

rng,

input=layer0_input,

image_shape=<batch_size,

1,

28,

28>,

filter_shape=<nkerns[0],

1,

5,

5>,

poolsize=<2,

2>

>

#layer1即第二個LeNetConvPoolLayer層#輸入是layer0的輸出,每張?zhí)卣鲌D為<12,12>,經(jīng)過conv得到<12-5+1,

12-5+1>

=

<8,

8>,#經(jīng)過maxpooling得到<8/2,

8/2>

=

<4,

4>#因為每個batch有batch_size張圖〔特征圖,第二個LeNetConvPoolLayer層有nkerns[1]個卷積核#,故layer1輸出為<batch_size,

nkerns[1],

4,

4>

layer1

=

LeNetConvPoolLayer<

rng,

input=layer0.output,

image_shape=<batch_size,

nkerns[0],

12,

12>,#輸入nkerns[0]張?zhí)卣鲌D,即layer0輸出nkerns[0]張?zhí)卣鲌D

filter_shape=<nkerns[1],

nkerns[0],

5,

5>,

poolsize=<2,

2>

>

#前面定義好了兩個LeNetConvPoolLayer〔layer0和layer1,layer1后面接layer2,這是一個全連接層,相當于MLP里面的隱含層#故可以用MLP中定義的HiddenLayer來初始化layer2,layer2的輸入是二維的<batch_size,

num_pixels>

,#故要將上層中同一張圖經(jīng)不同卷積核卷積出來的特征圖合并為一維向量,#也就是將layer1的輸出<batch_size,

nkerns[1],

4,

4>flatten為<batch_size,

nkerns[1]*4*4>=<500,800>,作為layer2的輸入。#<500,800>表示有500個樣本,每一行代表一個樣本。layer2的輸出大小是<batch_size,n_out>=<500,500>

layer2_input

=

layer1.output.flatten<2>

layer2

=

HiddenLayer<

rng,

input=layer2_input,

n_in=nkerns[1]

*

4

*

4,

n_out=500,

activation=T.tanh

>

#最后一層layer3是分類層,用的是邏輯回歸中定義的LogisticRegression,#layer3的輸入是layer2的輸出<500,500>,layer3的輸出就是<batch_size,n_out>=<500,10>

layer3

=

LogisticRegression<input=layer2.output,

n_in=500,

n_out=10>

#代價函數(shù)NLL

cost

=

layer3.negative_log_likelihood<y>

#

test_model計算測試誤差,x、y根據(jù)給定的index具體化,然后調(diào)用layer3,#layer3又會逐層地調(diào)用layer2、layer1、layer0,故test_model其實就是整個CNN結(jié)構(gòu),#test_model的輸入是x、y,輸出是layer3.errors<y>的輸出,即誤差。

test_model

=

theano.function<

[index],

layer3.errors<y>,

givens={

x:

test_set_x[index

*

batch_size:

<index

+

1>

*

batch_size],

y:

test_set_y[index

*

batch_size:

<index

+

1>

*

batch_size]

}

>

#validate_model,驗證模型,分析同上。

validate_model

=

theano.function<

[index],

layer3.errors<y>,

givens={

x:

valid_set_x[index

*

batch_size:

<index

+

1>

*

batch_size],

y:

valid_set_y[index

*

batch_size:

<index

+

1>

*

batch_size]

}

>

#下面是train_model,涉及到優(yōu)化算法即SGD,需要計算梯度、更新參數(shù)#參數(shù)集

params

=

layer3.params

+

layer2.params

+

layer1.params

+

layer0.params

#對各個參數(shù)的梯度

grads

=

T.grad<cost,

params>

#因為參數(shù)太多,在updates規(guī)則里面一個一個具體地寫出來是很麻煩的,所以下面用了一個for..in..,自動生成規(guī)則對<param_i,

param_i

-

learning_rate

*

grad_i>

updates

=

[

<param_i,

param_i

-

learning_rate

*

grad_i>

for

param_i,

grad_i

in

zip<params,

grads>

]

#train_model,代碼分析同test_model。train_model里比test_model、validation_model多出updates規(guī)則

train_model

=

theano.function<

[index],

cost,

updates=updates,

givens={

x:

train_set_x[index

*

batch_size:

<index

+

1>

*

batch_size],

y:

train_set_y[index

*

batch_size:

<index

+

1>

*

batch_size]

}

>

################

開始訓練

################print'...

training'

patience

=

10000

patience_increase

=

2

improvement_threshold

=

0.995

validation_frequency

=

min<n_train_batches,

patience

/

2>

#這樣設置validation_frequency可以保證每一次epoch都會在驗證集上測試。

best_validation_loss

=

numpy.inf

#最好的驗證集上的loss,最好即最小

best_iter

=

0#最好的迭代次數(shù),以batch為單位。比如best_iter=10000,說明在訓練完第10000個batch時,達到best_validation_loss

test_score

=

0.

start_time

=

time.clock<>

epoch

=

0

done_looping

=

False#下面就是訓練過程了,while循環(huán)控制的時步數(shù)epoch,一個epoch會遍歷所有的batch,即所有的圖片。#for循環(huán)是遍歷一個個batch,一次一個batch地訓練。for循環(huán)體里會用train_model<minibatch_index>去訓練模型,#train_model里面的updatas會更新各個參數(shù)。#for循環(huán)里面會累加訓練過的batch數(shù)iter,當iter是validation_frequency倍數(shù)時則會在驗證集上測試,#如果驗證集的損失this_validation_loss小于之前最佳的損失best_validation_loss,#則更新best_validation_loss和best_iter,同時在testset上測試。#如果驗證集的損失this_validation_loss小于best_validation_loss*improvement_threshold時則更新patience。#當達到最大步數(shù)n_epoch時,或者patience<iter時,結(jié)束訓練while

<epoch

<

n_epochs>

and

<not

done_looping>:

epoch

=

epoch

+

1for

minibatch_index

in

xrange<n_train_batches>:

iter

=

<epoch

-

1>

*

n_train_batches

+

minibatch_index

if

iter

%

100

==

0:

print'training

@

iter

=

',

iter

cost_ij

=

train_model<minibatch_index>

#cost_ij

沒什么用,后面都沒有用到,只是為了調(diào)用train_model,而train_model有返回值if

<iter

+

1>

%

validation_frequency

==

0:

#

compute

zero-one

loss

on

validation

set

validation_losses

=

[validate_model<i>

for

i

in

xrange<n_valid_batches>]

this_validation_loss

=

numpy.mean<validation_losses>

print<'epoch

%i,

minibatch

%i/%i,

validation

error

%f

%%'

%

<epoch,

minibatch_index

+

1,

n_train_batches,

this_validation_loss

*

100.>>

if

this_validation_loss

<

best_validation_loss:

if

this_validation_loss

<

best_validation_loss

*

\

improvement_threshold:

patience

=

max<patience,

iter

*

patience_increase>

best_validation_loss

=

this_validation_loss

best_iter

=

iter

test_losses

=

[

test_model<i>

for

i

in

xrange<n_test_batches>

]

test_score

=

numpy.mean<test_losses>

print<<'

epoch

%i,

minibatch

%i/%i,

test

error

of

''best

model

%f

%%'>

%

<epoch,

minibatch_index

+

1,

n_train_batches,

test_score

*

100.>>

if

patience

<=

iter:

done_looping

=

Truebreak

end_time

=

time.clock<>

print<'Optimization

complete.'>

print<'Best

validation

score

of

%f

%%

obtained

at

iteration

%i,

''with

test

performance

%f

%%'

%

<best_validation_loss

*

100.,

best_iter

+

1,

test_score

*

100.>>

print

>>

sys.stderr,

<'The

code

for

file

'

+

os.path.split<__file__>[1]

+

'

ran

for

%.2fm'

%

<<end_time

-

start_time>

/

60.>>

/tutorial/contents.htmlimporttheanofromtheanoimporttensorasTfromimportconv2dimportnumpyrng=numpy.random.RandomState<23455>#instantiate4Dtensorforinputinput=T.tensor4<name='input'>#initializesharedvariableforweights.w_shp=<2,3,9,9>w_bound=numpy.sqrt<3*9*9>W=theano.shared<numpy.asarray<rng.uniform<low=-1.0/w_bound,high=1.0/w_bound,size=w_shp>,dtype=input.dtype>,name='W'>#initializesharedvariableforbias<1Dtensor>withrandomvalues#IMPORTANT:biasesareusuallyinitializedtozero.Howeverinthis#particularapplication,wesimplyapplytheconvolutionallayerto#animagewithoutlearningtheparameters.Wethereforeinitialize#themtorandomvaluesto"simulate"learning.b_shp=<2,>b=theano.shared<numpy.asarray<rng.uniform<low=-.5,high=.5,size=b_shp>,dtype=input.dtype>,name='b'>#buildsymbolicexpressionthatcomputestheconvolutionofinputwithfiltersinwconv_out=conv2d<input,W>#buildsymbolicexpressiontoaddbiasandapplyactivationfunction,i.e.produceneuralnetlayeroutput#Afewwordson``dimshuffle``:#``dimshuffle``isapowerfultoolinreshapingatensor;#whatitallowsyoutodoistoshuffledimensionaround#butalsotoinsertnewonesalongwhichthetensorwillbe#broadcastable;#dimshuffle<'x',2,'x',0,1>#Thiswillworkon3dtensorswithnobroadcastable#dimensions.Thefirstdimensionwillbebroadcastable,#thenwewillhavethethirddimensionoftheinputtensoras#thesecondoftheresultingtensor,etc.Ifthetensorhas#shape<20,30,40>,theresultingtensorwillhavedimensions#<1,40,1,20,30>.<AxBxCtensorismappedto1xCx1xAxBtensor>#Moreexamples:#dimshuffle<'x'>->makea0d<scalar>intoa1dvector#dimshuffle<0,1>->identity#dimshuffle<1,0>->invertsthefirstandseconddimensions#dimshuffle<'x',0>->makearowoutofa1dvector<Nto1xN>#dimshuffle<0,'x'>->makeacolumnoutofa1dvector<NtoNx1>#dimshuffle<2,0,1>->AxBxCtoCxAxB#dimshuffle<0,'x',1>->AxBtoAx1xB#dimshuffle<1,'x',0>->AxBtoBx1xAoutput=T.nnet.sigmoid<conv_out+b.dimshuffle<'x',0,'x','x'>>#createtheanofunctiontocomputefilteredimagesf=theano.function<[input],output>fromimportpoolinput=T.dtensor4<'input'>maxpool_shape=<2,2>pool_out=pool.pool_2d<input,maxpool_shape,ignore_border=True>f=theano.function<[input],pool_out>invals=numpy.random.RandomState<1>.rand<3,2,5,5>print'Withignore_bordersettoTrue:'print'invals[0,0,:,:]=\n',invals[0,0,:,:]print'output[0,0,:,:]=\n',f<invals>[0,0,:,:]pool_out=pool.pool_2d<input,maxpool_shape,ignore_border=False>f=theano.function<[input],pool_out>print'Withignore_bordersettoFalse:'print'invals[1,0,:,:]=\n',invals[1,0,:,:]print'output[1,0,:,:]=\n',f<invals>[1,0,:,:]請注意,術(shù)語"卷積"可以對應于不同的數(shù)學運算：,這是幾乎所有的最近發(fā)表的卷積模型最常用的一個。在該操作中,每個輸出特征映射通過不同的2d濾波器連接到每個輸入特征映射,其值是通過相應濾波器的所有輸入的單個卷積的和。在原來的LeNet模型的卷積：在這項工作中,每個輸出特征映射只能連接到輸入特征映射的一個子集。用于信號處理的卷積：,它只適用于單通道輸入。在這里,我們使用的第一個操作,所以這個模型略有不同,從原來的LeNet研究。使用2的原因之一。將減少所需的計算量,但現(xiàn)代硬件使其具有完全連接模式的快速性。另一個原因是稍微減少自由參數(shù)的數(shù)量,但是我們還有其他的正則化技術(shù)。classLeNetConvPoolLayer<object>:"""PoolLayerofaconvolutionalnetwork"""def__init__<self,rng,input,filter_shape,image_shape,poolsize=<2,2>>:"""AllocateaLeNetConvPoolLayerwithsharedvariableinternalparameters.:paramrng:arandomnumbergeneratorusedtoinitializeweights:paraminput:symbolicimagetensor,ofshapeimage_shape:typefilter_shape:tupleorlistoflength4:paramfilter_shape:<numberoffilters,numinputfeaturemaps,filterheight,filterwidth>:typeimage_shape:tupleorlistoflength4:paramimage_shape:<batchsize,numinputfeaturemaps,imageheight,imagewidth>:typepoolsize:tupleorlistoflength2:parampoolsize:thedownsampling<pooling>factor<#rows,#cols>"""assertimage_shape[1]==filter_shape[1]self.input=input#thereare"numinputfeaturemaps*filterheight*filterwidth"#inputstoeachhiddenunitfan_in=d<filter_shape[1:]>#eachunitinthelowerlayerreceivesagradientfrom:#"numoutputfeaturemaps*filterheight*filterwidth"/#poolingsizefan_out=<filter_shape[0]*d<filter_shape[2:]>//d<poolsize>>#initializeweightswithrandomweightsW_bound=numpy.sqrt<6./<fan_in+fan_out>>self.W=theano.shared<numpy.asarray<rng.uniform<low=-W_bound,high=W_bound,size=filter_shape>,dtype=theano.config.floatX>,borrow=True>#thebiasisa1Dtensor--onebiasperoutputfeaturemapb_values=numpy.zeros<<filter_shape[0],>,dtype=theano.config.floatX>self.b=theano.shared<value=b_values,borrow=True>#convolveinputfeaturemapswithfiltersconv_out=conv2d<input=input,filters=self.W,filter_shape=filter_shape,input_shape=image_shape>#pooleachfeaturemapindividually,usingmaxpoolingpooled_out=pool.pool_2d<input=conv_out,ds=poolsize,ignore_border=True>#addthebiasterm.Sincethebiasisavector<1Darray>,wefirst#reshapeittoatensorofshape<1,n_filters,1,1>.Eachbiaswill#thusbebroadcastedacrossmini-batchesandfeaturemap#width&heightself.output=T.tanh<pooled_out+self.b.dimshuffle<'x',0,'x','x'>>#storeparametersofthislayerself.params=[self.W,self.b]#keeptrackofmodelinputself.input=input注意,在初始化權(quán)重值時,扇入取決于接收字段的大小和輸入特征映射的數(shù)量。最后,利用Logistic回歸和多層感知器的分類MNIST數(shù)字隱含定義類定義的回歸類,我們可以實例化網(wǎng)絡如下。x=T.matrix<'x'>#thedataispresentedasrasterizedimagesy=T.ivector<'y'>#thelabelsarepresentedas1Dvectorof#[int]labels#######################BUILDACTUALMODEL#######################print<'...buildingthemodel'>#Reshapematrixofrasterizedimagesofshape<batch_size,28*28>#toa4Dtensor,compatiblewithourLeNetConvPoolLayer#<28,28>isthesizeofMNISTimages.layer0_input=x.reshape<<batch_size,1,28,28>>#Constructthefirstconvolutionalpoolinglayer:#filteringreducestheimagesizeto<2