數(shù)據(jù)挖掘4章節(jié)挖掘原語(yǔ)語(yǔ)言和體系結(jié)構(gòu)ppt課件

1、發(fā)掘原語(yǔ)，言語(yǔ)和體系構(gòu)造n數(shù)據(jù)發(fā)掘原語(yǔ)n數(shù)據(jù)發(fā)掘言語(yǔ)n數(shù)據(jù)發(fā)掘系統(tǒng)體系構(gòu)造n總結(jié)數(shù)據(jù)發(fā)掘原語(yǔ)劃分發(fā)掘相關(guān)的數(shù)據(jù)發(fā)掘相關(guān)的數(shù)據(jù) 發(fā)掘的知識(shí)類型發(fā)掘的知識(shí)類型背景知識(shí)背景知識(shí)方式的興趣度量方式的興趣度量結(jié)果的表示與可視化結(jié)果的表示與可視化義務(wù)相關(guān)數(shù)據(jù)義務(wù)相關(guān)數(shù)據(jù)n數(shù)據(jù)庫(kù)或數(shù)據(jù)倉(cāng)庫(kù)稱號(hào)數(shù)據(jù)庫(kù)或數(shù)據(jù)倉(cāng)庫(kù)稱號(hào)n例如：例如：AllElectronics_dbn數(shù)據(jù)庫(kù)表或數(shù)據(jù)倉(cāng)庫(kù)的立方體數(shù)據(jù)庫(kù)表或數(shù)據(jù)倉(cāng)庫(kù)的立方體n例如：表例如：表item,customer,purchase,items_soldn數(shù)據(jù)選擇條件數(shù)據(jù)選擇條件n例如：選取本年度加拿大地域購(gòu)買商品的數(shù)據(jù)例如：選取本年度加拿大地域購(gòu)買商品的數(shù)據(jù)n選取

2、條件能夠在概念上層次高于選取條件能夠在概念上層次高于DB/DW的數(shù)據(jù)的數(shù)據(jù)n如：如：type=home entertainment，DB/DW中數(shù)據(jù)中數(shù)據(jù)tv,cd player,vcrn有關(guān)的屬性或維有關(guān)的屬性或維n例如：例如：item表的表的name,price屬性屬性;customer表的表的income,age屬性。屬性。n系統(tǒng)應(yīng)具備自動(dòng)選取相關(guān)屬性的機(jī)制，比如經(jīng)過(guò)評(píng)價(jià)各屬性與系統(tǒng)應(yīng)具備自動(dòng)選取相關(guān)屬性的機(jī)制，比如經(jīng)過(guò)評(píng)價(jià)各屬性與特定操作的相關(guān)程度。特定操作的相關(guān)程度。n數(shù)據(jù)分組的規(guī)范數(shù)據(jù)分組的規(guī)范n例如：根據(jù)日期進(jìn)展分組例如：根據(jù)日期進(jìn)展分組發(fā)掘的知識(shí)類型發(fā)掘的知識(shí)類型n描畫(huà)描畫(huà)(c

3、haracterization)n區(qū)別分析區(qū)別分析(discrimination)n關(guān)聯(lián)關(guān)聯(lián)(association)n分類分類/預(yù)測(cè)預(yù)測(cè)(classification/prediction)n聚類聚類(clustering)n例例:n用戶假設(shè)想開(kāi)掘用戶假設(shè)想開(kāi)掘AllElectronics數(shù)據(jù)庫(kù)中用戶數(shù)據(jù)庫(kù)中用戶的購(gòu)買習(xí)慣，能夠會(huì)選擇下面關(guān)聯(lián)規(guī)那么：的購(gòu)買習(xí)慣，能夠會(huì)選擇下面關(guān)聯(lián)規(guī)那么：nP(X:customer,W)Q(X,Y)=buys(X,Z)n X是是customer表的主鍵，表的主鍵，P,Q是謂詞變量是謂詞變量(在相在相關(guān)數(shù)據(jù)中定義關(guān)數(shù)據(jù)中定義)，W,Y,Z是目的變量。能夠的發(fā)掘結(jié)

4、果是：是目的變量。能夠的發(fā)掘結(jié)果是：nage(X,3039) income (X,40k49k) = buys(X,VCR)2.2%,60%naccupation(X,student)age(X,2029)=buys(X,computer)1.4%,70%背景知識(shí)：概念層次背景知識(shí)：概念層次n概念層次概念層次n方式層次方式層次(schema hierarchy)(schema hierarchy)n例：例：Streetcityprovince_or_statecountryStreetcityprovince_or_statecountryn集合集合- -分組層次分組層次(set-groupi

5、ng hierarchy)(set-grouping hierarchy)n例：例： young,middle_aged,seniorall(age) young,middle_aged,seniorall(age) n20-39 = young, 40-59 = middle_aged20-39 = young, 40-59 = middle_agedn基于操作層次基于操作層次(operation-derived hierarchy)(operation-derived hierarchy)n包括信息解碼，復(fù)雜數(shù)據(jù)對(duì)象的信息提取，數(shù)據(jù)聚類，數(shù)據(jù)分包括信息解碼，復(fù)雜數(shù)據(jù)對(duì)象的信息提取，數(shù)據(jù)聚類

6、，數(shù)據(jù)分布分析算法等布分析算法等n例：例： address: login-name department university address: login-name department university countrycountryn基于規(guī)那么層次基于規(guī)那么層次(rule-based hierarchy)(rule-based hierarchy)n例：例： low_profit_margin (X) = price(X, P1) and cost (X, low_profit_margin (X) = price(X, P1) and cost (X, P2) and (P1 - P

7、2) $50P2) and (P1 - P2) B)=n(A nd support (association),s(A=B)=n(A nd B)/n(all), B)/n(all), nnoise threshold (description)noise threshold (description)n新穎程度新穎程度(novelty)(novelty)n如：如：not previously known, surprising (used to not previously known, surprising (used to remove redundant rules, e.g., Can

8、ada vs. remove redundant rules, e.g., Canada vs. Vancouver rule implication support ratioVancouver rule implication support ratio結(jié)果方式的可視化結(jié)果方式的可視化n發(fā)掘系統(tǒng)應(yīng)可以用多種方式來(lái)顯示開(kāi)掘發(fā)掘系統(tǒng)應(yīng)可以用多種方式來(lái)顯示開(kāi)掘出來(lái)的方式出來(lái)的方式n如：規(guī)那么，表，報(bào)表，圖表，圖，決如：規(guī)那么，表，報(bào)表，圖表，圖，決策數(shù)和立方體策數(shù)和立方體n發(fā)掘系統(tǒng)應(yīng)可以支持發(fā)掘結(jié)果的多種操發(fā)掘系統(tǒng)應(yīng)可以支持發(fā)掘結(jié)果的多種操作作n如：如：drill-down , roll-up

9、, slicing , dicing ,rotation發(fā)掘原語(yǔ)，言語(yǔ)和體系構(gòu)造n數(shù)據(jù)發(fā)掘原語(yǔ)n數(shù)據(jù)發(fā)掘言語(yǔ)n數(shù)據(jù)發(fā)掘系統(tǒng)體系構(gòu)造n總結(jié)DMQL一個(gè)數(shù)據(jù)發(fā)掘言語(yǔ)n動(dòng)機(jī)動(dòng)機(jī)n為了能提供交互式數(shù)據(jù)發(fā)掘才干為了能提供交互式數(shù)據(jù)發(fā)掘才干n經(jīng)過(guò)提供一個(gè)類似經(jīng)過(guò)提供一個(gè)類似SQL的言語(yǔ)的言語(yǔ)n希望能像希望能像SQL言語(yǔ)一樣成為發(fā)掘規(guī)范言語(yǔ)言語(yǔ)一樣成為發(fā)掘規(guī)范言語(yǔ)n希望成為系統(tǒng)開(kāi)發(fā)和演化希望成為系統(tǒng)開(kāi)發(fā)和演化(evolution)的根底的根底n希望促進(jìn)信息交換，技術(shù)轉(zhuǎn)移，商業(yè)化并獲得希望促進(jìn)信息交換，技術(shù)轉(zhuǎn)移，商業(yè)化并獲得廣泛成認(rèn)廣泛成認(rèn)n設(shè)計(jì)設(shè)計(jì)nDMQL 在前面引見(jiàn)的發(fā)掘原語(yǔ)根底之上進(jìn)展設(shè)在前面引見(jiàn)的發(fā)

10、掘原語(yǔ)根底之上進(jìn)展設(shè)計(jì)計(jì)義務(wù)相關(guān)數(shù)據(jù)的語(yǔ)法表示義務(wù)相關(guān)數(shù)據(jù)的語(yǔ)法表示nuse database , or nuse data warehouse nfrom where nin relevance to norder by ngroup by nhaving 義務(wù)相關(guān)數(shù)據(jù)語(yǔ)法表示續(xù)義務(wù)相關(guān)數(shù)據(jù)語(yǔ)法表示續(xù)n例：假設(shè)發(fā)掘例：假設(shè)發(fā)掘AllElectronics的加拿大顧客經(jīng)常購(gòu)買的加拿大顧客經(jīng)常購(gòu)買的商品之間的關(guān)聯(lián)，針對(duì)顧客不同收入和年齡，并且的商品之間的關(guān)聯(lián)，針對(duì)顧客不同收入和年齡，并且數(shù)據(jù)用購(gòu)買日期進(jìn)展分組。發(fā)掘相關(guān)數(shù)據(jù)可以寫(xiě)成：數(shù)據(jù)用購(gòu)買日期進(jìn)展分組。發(fā)掘相關(guān)數(shù)據(jù)可以寫(xiě)成：nuse data

11、base AllElectronics_dbnin relevance to I.name , I.price , C.income , C.agenfrom customer C , item I , purchase P , item_sold Snwhere I.item_ID=S.item_ID and S.trans_ID=P.trans_ID and P.cust_ID=C.cust_IDngroup by P.daten發(fā)掘知識(shí)類型的語(yǔ)法發(fā)掘知識(shí)類型的語(yǔ)法n:= | | | | n:= Mine characterization as analyze n例：mine charac

12、teristics as customerPurchasing analyze count%n:=Mine comparison as for where versus where analyze n例：mine comparison as purchaseGroups for bigSpenders where avg(I.price) $100 analyze count versus budgetSpenders where avg(I.price),$100發(fā)掘知識(shí)類型的語(yǔ)法發(fā)掘知識(shí)類型的語(yǔ)法(續(xù)續(xù))n:=mine association as matching n例：mine ass

13、ociations as buyingHabits nmatching P(X:customer,W)Q(X,Y)=buys(X,Z)n:=mine classification as nanalyze n例： mine classification as classifyingCustomerCreditRating nanalyze credit_infon:=Mine prediction as nanalyze nset attribute_or_dimention_i=n例：mine prediction as predictItemPrice analyze price nset

14、category = “TV and brand=“SONY概念層次語(yǔ)法n語(yǔ)法：語(yǔ)法：nUse hierarchy for n不同概念層次采用不同定義方式不同概念層次采用不同定義方式n方式概念層次方式概念層次ndefine hierarchy time_hierarchy on date as date,month quarter,yearn集合集合-分組概念層次分組概念層次ndefine hierarchy age_hierarchy for age on customer asnlevel1: young, middle_aged, senior level0: allnlevel2: 2

15、0, ., 39 level1: youngnlevel2: 40, ., 59 level1: middle_agednlevel2: 60, ., 89 level1: senior概念層次語(yǔ)法續(xù)n基于操作概念方式基于操作概念方式(operation-derived hierarchies)ndefine hierarchy age_hierarchy for age on customer as n age_category(1), ., age_category(5) := cluster(default, age, 5) all(age)n基于規(guī)那么概念方式基于規(guī)那么概念方式(rul

16、e-based hierarchies)ndefine hierarchy profit_margin_hierarchy on item as n level_1: low_profit_margin level_0: alln if (price - cost) $50n level_1: medium-profit_margin $50) and (price - cost) = $250) n level_1: high_profit_margin $250興趣度量語(yǔ)法n語(yǔ)法：語(yǔ)法：nwith threshold = threshold_valuen例例:nwith support t

17、hreshold = 0.05nwith confidence threshold = 0.7 發(fā)掘知識(shí)表示語(yǔ)法n 用戶指定顯示方法用戶指定顯示方法ndisplay as display as n 為在不同概念層次上察看結(jié)果：為在不同概念層次上察看結(jié)果：nMultilevel_Manipulation := Multilevel_Manipulation := n roll up on roll up on | drill down on | drill down on | add | add | drop | drop 一個(gè)完好的DMQL語(yǔ)句use database AllElectroni

18、cs_db use hierarchy location_hierarchy for B.addressmine characteristics as customerPurchasing analyze count% in relevance to C.age , I.type , I.place_made from customer C, item I , purchases P , items_sold S , works_at W , branch Bwhere I.item_ID = S.item_ID and S.trans_ID = P.trans_ID and P.cust_I

19、D = C.cust_ID and P.method_paid = AmEx and P.empl_ID = W.empl_ID and W.branch_ID = B.branch_ID and B.address = Canada and I.price = 100with noise threshold = 0.05 display as table其它數(shù)據(jù)發(fā)掘言語(yǔ)n關(guān)聯(lián)規(guī)那么言語(yǔ)關(guān)聯(lián)規(guī)那么言語(yǔ)nMSQL (Imielinski & Virmani99)nMineRule (Meo Psaila and Ceri96) nQuery flocks 基于基于Datalog 語(yǔ)法語(yǔ)法

20、 (Tsur et al98)nOLEDB for DM (Microsoft2000)n和和 OLE DB, OLE DB for OLAP一同努力于一同努力于DB,DW,DM的規(guī)范的規(guī)范化化n到到2000年年3月止，曾經(jīng)完成了月止，曾經(jīng)完成了predictive modeling( classification & Prediction ), clustering,還未包還未包含含 characterization, discrimination , association modeling 等。等。nCRISP-DM (CRoss-Industry Standard Process for Data Mining)n是一個(gè)國(guó)際性工程，包含數(shù)據(jù)庫(kù)公司，數(shù)據(jù)倉(cāng)庫(kù)公司，用戶公司是一個(gè)國(guó)際性工程，包含數(shù)據(jù)庫(kù)公司，數(shù)據(jù)倉(cāng)庫(kù)公司，用戶公司(user companies)n目的在于提供有效數(shù)據(jù)發(fā)掘的平臺(tái)和過(guò)程構(gòu)造目的在于提供有效數(shù)據(jù)發(fā)掘的平臺(tái)和過(guò)程構(gòu)造(process structure)n強(qiáng)調(diào)運(yùn)用數(shù)據(jù)發(fā)掘技術(shù)來(lái)處理商業(yè)問(wèn)題強(qiáng)調(diào)運(yùn)用數(shù)據(jù)發(fā)掘技術(shù)來(lái)處理商業(yè)問(wèn)題發(fā)掘原語(yǔ)，言語(yǔ)和體系構(gòu)造n數(shù)據(jù)發(fā)掘原語(yǔ)n數(shù)據(jù)發(fā)掘言語(yǔ)n數(shù)據(jù)發(fā)掘系統(tǒng)體系構(gòu)造n總結(jié)數(shù)據(jù)發(fā)掘系統(tǒng)體系構(gòu)造n數(shù)據(jù)發(fā)掘系統(tǒng)與數(shù)據(jù)發(fā)掘系統(tǒng)與 DB/DW 系統(tǒng)的耦合程度系統(tǒng)的耦合程度n零耦合零耦合用文件作為數(shù)據(jù)源和存放結(jié)