信息檢索模型

1、An IR Model defines- 文檔的中間表示模型- 查詢的中間表示模型- 文檔與查詢的相關(guān)程度的計算機制QueriesDocumentsSimilarity ComputationlD 一組文檔的中間表示lQ 一組代表用戶信息需求的中間表示 (aka queries)lF 文檔建模框架：文檔、查詢及其關(guān)系lR(d,q) ranking 函數(shù)為D中的每個d和Q中的每個q分配一個實數(shù)定義關(guān)于q的一組文檔的有序關(guān)系l經(jīng)典模型布爾模型 : 集合理論lFuzzylExtended Boolean向量模型: 代數(shù)方法lGeneralised vectorlLatent semantic ind

2、exlNeural networks概率模型lInference networklBelief networkl結(jié)構(gòu)化模型Non-overlapping listsProximal nodeslRetrieve documents which are “true” for the queryl查詢 : 一組索引詞的邏輯組合lQ = (K1 K2) (K3 ( K4)l“檢索所有含K1和K2，或含K3但不含 K4的文檔”l設(shè)倒排文件集合為 D1,D2,D3,D4lK1 list : D1, D2, D3, D4lK2 list: D1, D2lK3 list: D1, D2, D3lK4 lis

3、t: D1Result: D1, D2, D3lK = K1, K2, K3, K4 是一組索引詞l文檔中所有索引詞權(quán)重為1。如果索引詞不在文檔中出現(xiàn)， weight = 0l則每個文檔用一個向量表示，每個索引詞一個權(quán)重.lD1 (1,1,1,1) lD2 (1,1,1,0)lD3 (1,0,1,0)lD4 (1,0,0,0)Query in disjunctive normal form(1,1,0,0) (1,1,0,1) (1,1,1,0), (1,1,1,1) (0,1,1,0) (0,0,1,0) (1,0,1,0)Result: D1, D2, D3lN 系統(tǒng)中索引詞總數(shù)lTi 索

4、引詞lT=t1,t2,tn 一組索引詞l文檔dj 中每個索引詞ki 的權(quán)重為 wi,j 0, wi,j = 0 if absentl則文檔 dj 可表示為一個文檔向量 dj = (w1,j, w2,j, , wnj)lgi 是一個函數(shù)，返回索引詞ti在任意文檔中的權(quán)重 gi(dj ) = wi,jl布爾向量: di ( or qi ) is set to 1 when the term ti is present in Document D (or in Query Q ) otherwise they are set to zerol所有的索引詞假定彼此獨立分布lA simplificati

5、onE.g. “Computer” and “network”l忽略詞之間的關(guān)系簡化了索引權(quán)重的計算，加快排序l用布爾表達式表示查詢有時并不容易和直觀l布爾判別標(biāo)準(zhǔn)文檔要幺相關(guān)，要幺不相關(guān)沒有中間地帶l布爾模型比較布爾查詢表達式和用于描述文檔內(nèi)容的一組索引詞l向量空間模型使用詞集來表示查詢和文檔，并計算二者之間的相似程度l文檔和查詢表示為高維空間中的點l空間中的每個維度與文檔集中的每個詞對應(yīng)d2q2d1q1Q q D di t2t1Two index termsn = 2Term 1Term 3Term 2D1D2D3D= (term1, term2, term3)l文檔與查詢越相關(guān)，則他們在

6、向量空間中的位置越近. l相似程度用向量關(guān)系來計算.l實際上是比較兩個向量的角度l向量中所有屬性的權(quán)和l點積l屬性向量的長度wi,ji = 1nwi,j wi,qi = 1ni = 1nw2i,j只是簡單計算那些使兩個向量分量都不為零的維度的個數(shù)Sim(dj, q) = | dj q | di * qi即共享索引詞的個數(shù).沒有考慮向量的大小Example: d1 = (2T1 , 3T2 , 5T3) d2 = (3T1 , 7T2 , 1T3)q = (0T1 , 0T2 , 2T3)Sim(d1 , q ) = 2*0 + 3*0 + 5*2 =10Sim(d2, q ) = 3*0 +

7、3*0 + 1*2 = 2| dj q |( | dj | | q | )Sim(dj, q) = Sim(dj, q) = wi,j wi,qi = 1n i = 1nw2i,ji = 1nw2i,q= cos Normalisation of the document and query vectors Example: d1 = (2T1 , 3T2 , 5T3)d2 = (3T1 , 7T2 , 1T3) q = (0T1 , 0T2 , 2T3)Sim(d1 , q ) = 10 / sqrt(4+9+25) x sqrt(4) = 10 / 6.16 x 2 = 0.81Sim(d

8、2, q ) = 2 / sqrt(9+49+2) x sqrt(4) = 2 / 7.74 x 2 = 0.12nuclearpeoplemachineludditecomputerpovertyunemploymentd1d2car0.220.330.20qd30.390.020.400.270.000.500.340.000.120.000.380.000.200.100.140.050.000.200.150.270.430.000.000.000.001.000.100.900.00Sim(d1,q) = 0.344Sim(d2,q) = 0.357Sim(d3,q) = 0.168

9、123Sim(dj, q) = wi,j wi,qi = 1n+ i = 1nwi,ji = 1nwi,qSim(dj, q) = wi,j wi,qi = 1n+ i = 1nwi,ji = 1nwi,q2i = 1nwi,jwi,q- DiceJaccardlDoc1 = (3,2,1,0,0,0,1,1)lDoc2 = (1,1,1,0,0,1,0,0)lDice = 2x(6) / (8+4) = 1lJaccard = 6/(8+4-6) = 1lCosine = 6/sqrt(16x4) = 6/8 = 0.75l加權(quán)提高了檢索性能 - easy to adapt to various weighting schemesl部分匹配允許近似匹配- easy to compute measur