求條件熵和互信息的matlab實現

本文格式為Word版，下載可任意編輯——求條件熵和互信息的matlab實現Assignment1_08116649_Chaoyun_Song

EEE315InformationTheoryand

CodingAssignment1

ChannelCapacityandMutual

Information

ID:08116649Name:Chaoyun.Song

Assignment1_08116649_Chaoyun_Song

1.Introduction

Shannon'sinformationcontentshouldhavesomeintuitiveproperties:1（（）Informationcontainedintheeventsoughttobedefinedintermsofsomemeasureofuncertaintyoftheevents.Lesscertaineventsoughttocontainmoreinformationthanmorecertainevents.Theinformationofunrelatedeventstakenasasingleeventshouldequalthesumoftheinformationoftheunrelatedevents.

Ininformationtheory,entropyisameasureoftheuncertaintyassociatedwitharandomvariable.Inthiscontext,thetermusuallyreferstotheShannonentropy,whichquantifiestheexpectedvalueoftheinformationcontainedinamessage,usuallyinunitssuchasbits.Inthiscontext,a'message'meansaspecificrealizationoftherandomvariable.Entropyisdefinedas:

H(x)???p(x)logp(x)itcanbeviewedas:ameasureoftheminimumcostneededtosendsomeformofinformation;“theamountofsurprisefactor〞oftheinformationmeasuredinbits.orhowmuchenergyitisworthspendingtocarrytheinformationwhichtranslatestotheminimumnumberofbitsneededtocodetheinformation.

Inprobabilitytheoryandinformationtheory,themutual

information(sometimesknownbythearchaictermtransinformation)oftworandomvariablesisaquantitythatmeasuresthemutualdependenceofthetworandomvariables.Themostcommonunitofmeasurementofmutualinformationisthebit,whenlogarithmstothebase2areused.Themutualinformationcanbedefinedas:

wherep(x,y)isthejointprobabilitydistributionfunctionofXandY,andp1(x)andp2(y)arethemarginalprobabilitydistributionfunctionsofXandYrespectively.

Assignment1_08116649_Chaoyun_Song

2.ResultwithMatlabstriptsandfunctions

(1).WriteaMatlabfunctiontocalculatetheentropyofasourcegivenadiscretedistribution.Cacluatetheentropyforthefollowingdistribution.Plottheentropydiagramforeachofthedistributions.A={B={

C={0.1,0.31,0.001,0.009,0.2,0.15,0.23}Solution:Matlabcode

>>A=[1/21/41/81/8];>>H1=-sum(A.*log2(A))H1=

1.7500

>>B=[1/41/41/41/4];>>H2=-sum(B.*log2(B))H2=2

>>C=[0.1,0.31,0.001,0.009,0.2,0.15,0.23];>>H3=-sum(C.*log2(C))

1111,,248814}

14,

14,

14,}

Assignment1_08116649_Chaoyun_Song

H3=

2.2897

(2).WriteaMATLAB[1]scripttoplotthecapacityofabinarysymmetricchannelwithcrossprobabilitypasfunctionofpwhere0

?p?1.Forwhatvalueofpisthecapacityminimizedandwhatis

theminimumvalue?

Forabinarysymmetricchannel(BSC),weknowthatP(0|1)=P(1|0)=p,P(0|0)=P(1|1)=1-p,which?p?isthecrossprobability0?p?1.WhenP(Y0)=P(Y1)=0.5themutualinformationcomestotheminimumvalue.Thecapacityofthischannelislike:

C=P(X0)P(0|0)log[P(0|0)/0.5]+P(X0)P(1|0)log[P(1|0)/0.5]+P(X1)P(0|1)log[P(0|1)/0.5]+P(X1)P(1|1)log[P(1|1)/0.5]=plog2p+(1-p)log2(1-p)+1

Usingmatlabwecanplotthediagramof?p?and?C?

Assignment1_08116649_Chaoyun_Song

Solution：MatlabCode：>>p=0:0.01:1;

>>C=p.*log2(p)+(1-p).*log2(1-p)+1;>>plot(p,C)

Fromthediagram,wecanseethechangeof?C?withdifferentvalueof?p?.Whenp=1thechannelcapacityis1(bit/symbol)

Whenp=0.5therearenoinformationandthemutualinformationis0When0.5?p?1thediagramissameasleftside