深度學(xué)習(xí)12優(yōu)化

optimization

優(yōu)化問(wèn)題梯度下降與隨機(jī)梯度下降代碼演示補(bǔ)充：自適應(yīng)學(xué)習(xí)率算法優(yōu)化問(wèn)題TheFirstPart無(wú)約束優(yōu)化問(wèn)題約束優(yōu)化問(wèn)題拉格朗日乘子法（純等式約束）KKT條件（混合不等式約束）梯度下降與隨機(jī)梯度下降TheSecondPartGD(GradientDescent)算法終止條件1.給定2.若終止條件滿足，則迭代終止3.令

4.一維精確線搜索求5.,轉(zhuǎn)第二步梯度方向步長(zhǎng)香蕉函數(shù)（Rosenbrock）

與數(shù)值優(yōu)化問(wèn)題不同，神經(jīng)網(wǎng)絡(luò)中的優(yōu)化算法通常是間接作用的。通過(guò)優(yōu)化代價(jià)函數(shù)J(θ)而不是直接純優(yōu)化目標(biāo)J本身。通過(guò)迭代求解最優(yōu)的權(quán)重問(wèn)題。將機(jī)器學(xué)習(xí)問(wèn)題轉(zhuǎn)化為優(yōu)化問(wèn)題即最小化訓(xùn)練集上的期望損失。神經(jīng)網(wǎng)絡(luò)中的優(yōu)化問(wèn)題SGD(Stochasticgradientdescent)

考慮最小化如下目標(biāo)函數(shù)

當(dāng)現(xiàn)實(shí)中樣本個(gè)數(shù)較多，且不存在簡(jiǎn)單計(jì)算公式時(shí)，依次計(jì)算每一個(gè)子函數(shù)的梯度就會(huì)非常耗時(shí)，為此我們每一步迭代只選擇一個(gè)子函數(shù)，將它梯度的負(fù)值作為迭代方向。SGD在神經(jīng)網(wǎng)絡(luò)中的算法帶動(dòng)量的SGD動(dòng)量v累積了上一次負(fù)梯度下降的信息，參數(shù)α決定了之前梯度的貢獻(xiàn)。類比于物理學(xué)中的慣性代碼演示TheThirdPartAssumption#-*-coding:utf-8-*-importrandom#Thisisasampletosimulateafunctiony=theta1*x1+theta2*x2#目標(biāo)函數(shù)input_x=[[1,4],[2,5],[5,1],[4,2]]y=[19,26,19,20]#訓(xùn)練集theta=[1,1]#待估參數(shù)loss=10#損失step_size=0.001#步長(zhǎng)eps=0.0001#終止條件max_iters=10000#最大步數(shù)error=0#偏差iter_count=0#步數(shù)統(tǒng)計(jì)while(loss>epsanditer_count<max_iters):loss=0

#這里更新權(quán)重的時(shí)候所有的樣本點(diǎn)都用上了

foriinrange(3):pred_y=theta[0]*input_x[i][0]+theta[1]*input_x[i][1]theta[0]=theta[0]-step_size*(pred_y-y[i])*input_x[i][0]theta[1]=theta[1]-step_size*(pred_y-y[i])*input_x[i][1]#梯度下降法調(diào)整參數(shù)foriinrange(3):pred_y=theta[0]*input_x[i][0]+theta[1]*input_x[i][1]error=0.5*(pred_y-y[i])**2loss=loss+error#重新計(jì)算損失iter_count+=1#步數(shù)更新print'iters_count',iter_countprint'theta:',thetaprint'finalloss:',lossprint'iters:',iter_counttheta:[3.0027765778748003,3.997918297015663]finalloss:9.68238055213e-05iters:225#-*-coding:utf-8-*-importrandom#Thisisasampletosimulateafunctiony=theta1*x1+theta2*x2#目標(biāo)函數(shù)input_x=[[1,4],[2,5],[5,1],[4,2]]y=[19,26,19,20]#訓(xùn)練集theta=[1,1]#待估參數(shù)loss=10#損失step_size=0.001#步長(zhǎng)eps=0.0001#終止條件max_iters=10000#最大步數(shù)error=0#偏差iter_count=0#步數(shù)統(tǒng)計(jì)while(loss>epsanditer_count<max_iters):loss=0

#每一次選取一個(gè)點(diǎn)進(jìn)行權(quán)重的更新

foriinrange(3):pred_y=theta[0]*input_x[i][0]+theta[1]*input_x[i][1]theta[0]=theta[0]-step_size*(pred_y-y[i])*input_x[i][0]theta[1]=theta[1]-step_size*(pred_y-y[i])*input_x[i][1]#梯度下降法調(diào)整參數(shù)foriinrange(3):pred_y=theta[0]*input_x[i][0]+theta[1]*input_x[i][1]error=0.5*(pred_y-y[i])**2loss=loss+error#重新計(jì)算損失iter_count+=1#步數(shù)更新print'iters_count',iter_countprint'theta:',thetaprint'finalloss:',lossprint'iters:',iter_counttheta:[3.001599022700168,3.9988067924856194]finalloss:9.76582893754e-05iters:3384#-*-coding:utf-8-*-importnumpyasnpN,D_in,H,D_out=64,1000,100,10#Nisbatchsize;D_inisinputdimension; Hishiddendimension;D_outisoutputdimension.#Createrandominputandoutputdatax=np.random.randn(N,D_in)y=np.random.randn(N,D_out)#標(biāo)準(zhǔn)正態(tài)分布#Randomlyinitializeweights#定義訓(xùn)練參數(shù)w1=np.random.randn(D_in,H)w2=np.random.randn(H,D_out)learning_rate=1e-6fortinrange(500):

#前向傳播，對(duì)y進(jìn)行預(yù)測(cè)h=x.dot(w1)#隱層h_relu=np.maximum(h,0)#relu激活函數(shù)y_pred=h_relu.dot(w2)#輸出層loss=0.5*np.square(y_pred-y).sum()#損失函數(shù)

#反向傳播grad_y_pred=(y_pred-y)grad_w2=h_relu.T.dot(grad_y_pred)grad_h_relu=grad_y_pred.dot(w2.T)grad_h=grad_h_relu.copy()grad_h[h<0]=0grad_w1=x.T.dot(grad_h)#鏈?zhǔn)角髮?dǎo)法則

#更新w1,w2w1-=learning_rate*grad_w1w2-=learning_rate*grad_w2