北大暑期課程《回歸分析》(Linear-Regression-Analysis)講義2

Class2:Basicsofmatrix(2)Linearregression.Basicsofmatrix(2)「InverseofaMatrix Theinverseofasquarematrix A(nn)existsifthematrixisnonsingular.TheinverseA-1isdefinedas:A-1A=AA-1=IAlternatively,theconditioncanbeexpressedinthreeotherforms:(1)Ahasrankn,(2)thenrowsarelinearlyindependent,and(3)thencolumnsarelinearlyindependent.Inverseisadifficultoperation.Usuallywecanusecomputersoftwarestofindtheinverse.Hereweonlywanttoknowasimpleexample.Fora2x2matrix:A-1dDbDA-1dDbDcDaDwhereDisthedeterminantofA.D(A)=ad-bc.DeterminantofaMatrixThedeterminantofamatrixisascale.Anonsingularmatrixhasanon-zerodeterminant.OperationRulesofMatrices A=Bmeans aij bijforalli,jA+B=B+A(A+B)+C=A+(B+C)(AB)C=A(BC)C(A+B)=CA+CBc(A+B)=cA+cB,wherecisascalarIA=AI=AA+O=AAO=OA=O(A')'=A(A+B)'=A'+B'(AB)'=B'A'(ABC)'=C'B'A'(AB)-1=B-1A-1,providedAandBareeachnonsingular(proof:ABB-1A-1=I)(ABC)-1=C-1B-1A-1(A-1)-1=A(A’-1=(A-1)，o,bo,b1,…bk),itsvariance-covariancematrixForavectorofvariablesbwithelements(bClass2,PageClass2,PageV(b)V(bo)Cov(bo,bi)V(bi)Cov(bo,bk) ... V(bk)LinearRegressionwithaSingleRegressor(SimpleRegression)Forsimplelinearregression,welearned,yi 0 1xi iWeassumethatthismodelistrueonlyinthepopulation.Whatwecanobserve,however,isasample.Forasampleoffixedsizei=1,…n,wecanwritethemodelinthefollowingway:⑴y0iX_where中y...ynXix...xn1nLetusfurtherassumethat1x1n2 ... 1x1Xnand01Equation(1)becomes⑵yXn221 -[expandfromthematrixformintotheelementform]

yi丫2yn...Pre-multiply⑵byX'⑶ (X'y)(X'X)_ (X'_)WesetX'eo(orthogonalitycondition),meaning(e) 0(firstelement);(xi0) 0(secondelement).Giventheorthogonalitycondition,wecaneasilysolvebas _1⑸b(X'X)1X'y,Whydoweassumetheorthogonalitycondition?Becauseorthogonalitygivestheleastsquaressolutionbestlinearpredictor.[Blackboard]2TOC\o"1-5"\h\zPartial yi (b0 biXi)withrespectto b0,b1,settozero. (e) 0, (0X) 0.Inpractice,wedon'tknowwhetherXsatisfiestheorthogonalitycondition.Weusuallymaketheassumption:()0(x) Cov(x,_) 0.Notethatthefirstassumptionmeansorthogonalitybetween1and .Thesecondassumptionmeansthatxisnotcorrelatedwith .1 x1 1 x1I.. ...1 xn 1 xn1x1\o"CurrentDocument"1 ... 1Similarly,x1nxi...xnxi2xi1 xny1...yn(X)2y1...yn(X)2n Xi2(XiX)2](Xi X)2]2 2 2n2(X)2n(xiX)2Xi/[n(X^X)2]n/[n(XiX)2]2n[(XiX)(yY)]/[n(xX)2]2[(XiX)(yiY)]/[ (XiX)2]b02 2x yi/[n(XX)2]2rXi Xiyi/[n(XiX)]... 1'y一 x1 ...xnV\XV2Det(')nX，Xi/[n(')Xi/[nLetussolveforbb(')1'y2 2-1bX yj[n(xX)]n XiyJ[n (Xi))]2(nxv n2XY)/[n(為X)2]TOC\o"1-5"\h\z2 2r[XyX Xiyi]/[nXX)]2 —2 —2 2[X yinXV\nXyxXyJ/[n(xX)]2 2[yi (XiX)2X(nX yinXiyJ/[n(XiX)2]yi/nX[nxiyin2XY]/[n(xiX)2]- 2YX(XiyinXY]/[(xX)2]2YX[(xX)(yiY)]/[(xX)]Y Xb1Thus,bisindeedyouroldfriend:

YbiX2(XiX)(yiY)/(XiX)2InferenceofRegressionCoefficients(simpleregression)Defineexpectationofavector:E(b)takeexpectationofeachoftheelements.Definevarianceofavector:V(b)isasymmetricmatrix,calledvarianceandcovariancematrixofb.V(b)V(b。)Cov(b0,b1)V(b)Cov(b0,b1)V(bJProperty:ifAisamatrixwithonlyconstantelements,E(Ab)AE(b)V(Ab)AV(b)A'TheLSEstimatorForthemodely X_,b(X'X)1X'yPropertiesoftheLSEstimator1E(b)E[(X'X1X'y]E[(X'X)1X'(X__)]1 1E[(X'X)1X'X」E[(X'X)1X'_]1 1(X'X)1X'XE[_](X'X)1E[X'_]thatis,bis_unbiased.1V(b)V[(X'X)1X'y]V[(X'X)1X'(X__)]V[(X'X)1X'X(X'X)1X'_)]1V[(X'X)X'_)]1(X'X)1X'V[」X(X'X)(afterassuming1 :(X'X)[blackboard]V[]2V[]I,non-senalcorrelationandhomoscedasticity)Wethenneednormalityassumptionforstatisticalinferences.Recalltheformula:V(b1)= 2/[(XiX)2]FittedValuesandResidualsyXbX(X'X)1X'yHyInterpretationofprojection[3-dgraph]HnnX(X'X)1X'iscalledHmatrix,orhatmatrix.Hisanidempotentmatrix:HHHForresiduals:ey _? yHy(IH)y(I-H)isalsoaidempotentmatrix.EstimationoftheResidualVariance A.SampleAnalog2V(i)E[iE(i)]2E[J2isunknownbutcanbeestimatedbye,whereeisresidual.SomeofyoumayhavenoticedthatIhaveintentionallydistinguished frome.iscalleddisturbance,andeiscalledresidual.Residualisdefinedbythedifferencebetweenobservedandpredictedvalues.Thesampleanalogof(6)is2ein1n1nyiyi?i2b0b1xi1b2xi2bp1xip1Inmatrix:20eeThesampleanalogisthene'e/nDegreesofFreedomLetusreviewbrieflytheconceptofdegreesoffreedom.Asageneralrule,thecorrectdegreesoffreedomequalsthenumberoftotalobservationsminusthenumberofparametersusedinestimation.Sinceweobtainresidualsafterweuseestimatedcoefficients,theresidualsaresubjecttolinearconstrained(recallorthogonalityconstraints).Forexample:Ifn=2,p=2,wehavethesaturatedmodel,e 1=0,e2=0.Ifn=3,p=2,thereisonly1degreeoffreedom.e 1=-e1Inmultipleregression,therearepparameterstobeestimated.