使用挖掘介紹及趨勢(shì)

2023/9/1WebUsageMining1提綱1.簡(jiǎn)介1.1WUM的定義和應(yīng)用1.2WUM基本的過(guò)程1.3原型系統(tǒng)和商用系統(tǒng)2.趨勢(shì)2023/9/1WebUsageMining2WebMining的分類[1][2]：1.ContentMining2.StructureMining3.UsageMiningWebMining分類2023/9/1WebUsageMining31.1.1定義(JaideepSrivastava[1])“theprocessofapplyingdataminingtechniquestothediscoveryofusagepatternsfromWebdata”2023/9/1WebUsageMining41.1.2功能SystemImprovement1).SiteImprovement2).WebCachingandNetworkTransmission(E.Cohen[5])2023/9/1WebUsageMining51.1.2功能1).SiteImprovement根據(jù)實(shí)際用戶的瀏覽情況，調(diào)整網(wǎng)站的網(wǎng)頁(yè)的連接結(jié)構(gòu)和內(nèi)容，更好的服務(wù)用戶A=>B=>C=>DA=>D極端：Adaptivewebsites(MikePerkowitz[7][8])2023/9/1WebUsageMining61.1.2功能2).Caching&NetworkTransmission(E.Cohen[5])例如：從proxy的訪問(wèn)信息中可以分析用戶的訪問(wèn)模式，從而可以預(yù)測(cè)用戶的Page訪問(wèn)，提高WebCaching的性能A=>B=>CA=>B=>DA=>BCached:C、D2023/9/1WebUsageMining71.1.2功能*與傳統(tǒng)的銷售途徑不同的是：網(wǎng)上零售系統(tǒng)可捕捉到大量的采購(gòu)過(guò)程的細(xì)節(jié)，提供了更加深入分析的可能2023/9/1WebUsageMining81.1.2功能Personalization定義：根據(jù)發(fā)現(xiàn)的用戶喜好，動(dòng)態(tài)地為用戶定制觀看的內(nèi)容 或提供瀏覽建議。直接實(shí)現(xiàn)形式：Recommender系統(tǒng)([10]J.BenSchafer)。作用：1)方便用戶查詢和瀏覽2)增強(qiáng)廣告的作用3)促進(jìn)網(wǎng)上銷售4)提高用戶忠誠(chéng)度2023/9/1WebUsageMining91.2基本的過(guò)程*1).DataGathering2).Preprocessing3).Patterndiscovery4).Patternsanalysis2023/9/1WebUsageMining101.2基本的過(guò)程SiteFiles,RawUsageData,UserProfileSiteContent&Structure,Server/Session/EpisodeDataUsageStatistics,UserClusters,AssociationRules,SequentialPatternsKnowledge1).DataGathering2).Preprocessing4).Patternsanalysis3).Patterndiscovery2023/9/1WebUsageMining111.2.1DataGathering可以利用的WebData包括:1).Content:頁(yè)面的實(shí)際內(nèi)容2).Structure:包括intra-structure和inter-structure3).Usage:例如：ClickStream4).UserProfile:例如：registrationdata和customerprofiles5.BusinessData？2023/9/1WebUsageMining121.2.1DataGathering數(shù)據(jù)來(lái)源:1)ServerLevelCollection2)ClientLevelCollection3)ProxyLevelCollection2023/9/1WebUsageMining131.2.1DataGatheringServerLevelCollection:WebLogFile2)PacketSniffing技術(shù)缺點(diǎn)是擴(kuò)展性差（加密通道和用戶跟蹤）3)WebPageContent&structure4)ApplicationServer2023/9/1WebUsageMining141.2.1DataGatheringClientLevelCollection:優(yōu)點(diǎn)是可以比較全面和準(zhǔn)確收集到用戶數(shù)據(jù)Applet&Script 缺點(diǎn):不支持代理，功能會(huì)被用戶關(guān)閉。Modifiedbrowser 優(yōu)點(diǎn)是全面和準(zhǔn)確收集到用戶數(shù)據(jù)，缺點(diǎn)是需要吸引或強(qiáng)制用戶使用2023/9/1WebUsageMining151.2.1DataGatheringProxyLevelCollection:這種數(shù)據(jù)收集方式適合有大量靜態(tài)頁(yè)面的網(wǎng)站2023/9/1WebUsageMining161.2.2Preprocessing1).Content&Structure2).UsageData2023/9/1WebUsageMining171.2.2PreprocessingContent&Structure:解決兩個(gè)問(wèn)題第一、page的獨(dú)特性的表示方法？第二、page的內(nèi)容和結(jié)構(gòu)的量化形式（quantifiableform）2023/9/1WebUsageMining181.2.2Preprocessingstructure舉例：ABCSiteMapEindex1AB1CFNavigationMixedMediaPageType2023/9/1WebUsageMining191.2.2PreprocessingABCSiteMap的結(jié)構(gòu)可表示為：M=<F1,F2,F3,F4,F5,F6,F7>F1={index,(frame,1,left|frame,A,main)}F2={1,(get,A,main),(get,C,main)}F3={A,(get,B,top)}F4={C,(get,E,top),(get,F,top)}F5={B}F6={E}F7={F}2023/9/1WebUsageMining201.2.2PreprocessingIndex=>1-A=>1-C=>EIndex=>1-A=>BMap的結(jié)構(gòu)表示了用戶可能的click訪問(wèn)的路徑2023/9/1WebUsageMining211.2.2PreprocessingContentClassification:1)UsageType2)ContentFeatures*具體種類的劃分跟應(yīng)用的環(huán)境相關(guān)2023/9/1WebUsageMining221.2.2PreprocessingUsageType1)HeadPage例如：Index2)MediaPage例如：B,E,F3)NavigationPage例如：1,C4)MixedPage例如：A2023/9/1WebUsageMining231.2.2PreprocessingContentFeatures([22]SoumenChakrabarti):1?？梢允褂胿ectorspacemodel來(lái)表示page的內(nèi)容：把page的示成一個(gè)在此高維空間中的一個(gè)向量page出現(xiàn)的term作為向量的維,維對(duì)應(yīng)的值是通過(guò)某種方式計(jì)算出來(lái)的權(quán)重(例如：TFIDF),它反映了term在此page區(qū)別其他page的時(shí)候的重要程度。2。Clustering&Classification2023/9/1WebUsageMining241.2.2PreprocessingUsageData:數(shù)據(jù)的抽象[4]1.User2.PageView(Click)

用戶在某個(gè)時(shí)刻看到的瀏覽器中的內(nèi)容3.ClickStream

一組連續(xù)的pageview請(qǐng)求4.UserSession(transaction)用戶的階段性的clickstream,可以跨多個(gè)服務(wù)器5.ServerSession

用戶的,在一個(gè)服務(wù)器上的階段性的clickstream6.Episode

用戶session中的一段有意義的clickstream2023/9/1WebUsageMining251.2.2PreprocessingUsageData:數(shù)據(jù)的抽象[4]RawDataPageViewClickStreamUserSessionServerSessionEpisodes2023/9/1WebUsageMining261.2.2PreprocessingUsagePreprocessing的流程1.數(shù)據(jù)清理2.識(shí)別User/Session3.識(shí)別PageView4.識(shí)別Episode2023/9/1WebUsageMining271.Datacleaning:*把log條目分割成對(duì)應(yīng)的數(shù)據(jù)項(xiàng)目*剔除圖片或其它非page類的log條目*剔除spider/agent的訪問(wèn)條目*規(guī)格化URI*提取POST數(shù)據(jù)項(xiàng)1.2.2Preprocessing2023/9/1WebUsageMining281.2.2Preprocessing清理后的SampleLogIPAddressTime/DateMethod/URIReferrerAgent15:30:01/2-Jan-01GETIndex.htm/link.htmMozilla/4.0(IE5.0W98)15:30:01/2-Jan-01GET1.htm/index.htmMozilla/4.0(IE5.0W98)15:30:01/2-Jan-01GETA.htm/index.htmMozilla/4.0(IE5.0W98)15:37:09/2-Jan-01GETE.htm/C.htmMozilla/4.0(IE5.0W98)15:33:04/2-Jan-01GETIndex.htm/res.phpMozilla/4.0(IE4.0NT)15:33:04/2-Jan-01GET1.htm/index.htmMozilla/4.0(IE4.0NT)15:33:04/2-Jan-01GETA.htm/index.htmMozilla/4.0(IE4.0NT)15:35:11/2-Jan-01GETB.htm/A.htmMozilla/4.0(IE4.0NT)15:35:11/2-Jan-01GETC.htm/A.htmMozilla/4.0(IE5.0W98)2023/9/1WebUsageMining292.UserandSessionidentification:1.2.2Preprocessing1.IPAddress&Agent2.EmbeddedSessionID3.Registration(UserProfile)4.Cookie5.SoftwareAgent(Applet&Scrtipt)6.ModifiedBrowser2023/9/1WebUsageMining30在只有IP/Agent的情況下，我們假設(shè)User和Session是等同*的:1.2.2Preprocessing15:33:04/2-Jan-01GETIndex.htm/res.php15:33:04/2-Jan-01GET1.htm/index.htm15:33:04/2-Jan-01GETA.htm/index.htm15:35:11/2-Jan-01GETB.htm/A.htm15:30:01/2-Jan-01GETIndex.htm/link.htm15:30:01/2-Jan-01GET1.htm/index.htm15:30:01/2-Jan-01GETA.htm/index.htm15:37:09/2-Jan-01GETE.htm/C.htm15:35:11/2-Jan-01GETC.htm/A.htmMozilla/4.0(IE5.0W98)User1:Mozilla/4.0(IE4.0NT)User2:2023/9/1WebUsageMining313.PageViewIdentification:1.2.2Preprocessing1-A/res.phpBA.htm1-A/link.htmEC.htm1-CA.htmMozilla/4.0(IE5.0W98)User1:Mozilla/4.0(IE4.0NT)User2:2023/9/1WebUsageMining32PathCompletion解決由于Cache帶來(lái)的問(wèn)題路徑不全的問(wèn)題1.2.2Preprocessingtime1-A1-C1-E1-C1-FSession_1Session_22023/9/1WebUsageMining334.EpisodeIdentification:1.2.2Preprocessing定義:meaningfulsubsetofuser/serversessionAuxiliary-MediaEpisodes:timeAAMMAAAMAEpisode1Episode2Episode32023/9/1WebUsageMining34Auxiliary-MediaEpisodes:1.2.2Preprocessing可以使用的方法包括：1）Pagetype2）Referencelength3）MaximalForwardReference2023/9/1WebUsageMining351).PagetypeEpisode:根據(jù)pageview的類型來(lái)確定1.2.2PreprocessingtimeAAMMAAAMAEpisode1Episode2Episode3SessionX缺點(diǎn)是，必須要求每個(gè)pageview有固定的用途。2023/9/1WebUsageMining362).ReferenceLengthEpisode:pageview的類型是根據(jù)用戶在此處停留的時(shí)間來(lái)確定。停留時(shí)間相對(duì)長(zhǎng)的，就認(rèn)為是MediaPage,停留時(shí)間短的則是一個(gè)AuxiliaryPage1.2.2Preprocessing2023/9/1WebUsageMining372).MaximalForwardReferenceEpisode[23,chen]它的假設(shè)是：一個(gè)episode是訪問(wèn)某個(gè)Mediapage的一個(gè)最大深度的pageview引用路徑，即從用戶session的第一個(gè)pageview開(kāi)始，一直向前訪問(wèn)，直到有一個(gè)backwardpageview出現(xiàn)1.2.2PreprocessingIndex1-A1-CE1-CBTimeEpisode1Episode22023/9/1WebUsageMining381.2.3PatternDiscovery其中有：1)StatisticalAnalysis2)FrequentItemsetsandAssociationRules3)Clustering&Classification4)SequentialPatterns5)DependencyModeling2023/9/1WebUsageMining391.2.3PatternDiscovery1)StatisticalAnalysis主要用于改進(jìn)系統(tǒng)的Performance,design等包括：1)mostfrequentlyaccessedpages 2)averageviewtimeofapage 3)averagelengthofapaththroughasite2023/9/1WebUsageMining401.2.3PatternDiscovery2)FrequentItem-setsandAssociationRules可以尋找出經(jīng)常頻繁訪問(wèn)的page組，可用于修改WebSite的設(shè)計(jì)或提前緩沖頁(yè)面，改進(jìn)系統(tǒng)的性能。2023/9/1WebUsageMining411.2.3PatternDiscovery3)Clustering&Classification包括兩方面的應(yīng)用：*user用于Marketsegmentation(市場(chǎng)分割)和個(gè)人內(nèi)容定制*page(content)后者主要用于IR和SurfAssistance2023/9/1WebUsageMining421.2.3PatternDiscovery5)SequentialPatterns可用于用戶的visitpattern.包括：1.trendanalysis2.changepointdetection3.similarityAnalysis.2023/9/1WebUsageMining431.2.3PatternDiscovery6)DependencyModeling不僅提供了理論化分析用戶行為的框架，同時(shí)也可以用來(lái)，提高網(wǎng)上產(chǎn)品銷量，提高用戶的訪問(wèn)提供方便可以使用的方法有：1）MarkovModel2）BayesianBriefNetwork2023/9/1WebUsageMining441.2.4PatternAnalysis目的是根據(jù)實(shí)際應(yīng)用，通過(guò)用戶的選擇和觀察，把發(fā)現(xiàn)的rules,patterns和statistics轉(zhuǎn)換為知識(shí)Knowledge。1）Informationfiltering(例如：SQL),2）Visualization3)OLAP4)KnowledgeQuery2023/9/1WebUsageMining451.3原型系統(tǒng)和商用系統(tǒng)2023/9/1WebUsageMining46IBM:SpeedTracer[17]從日志文件中重建usertraversalpath,然后識(shí)別出usersession;在此基礎(chǔ)上，利用數(shù)據(jù)挖掘算法發(fā)現(xiàn)mostcommontraversalpath和frequentlyvisitedpagegroup;系統(tǒng)提供三種統(tǒng)計(jì)類型的報(bào)告：1)user-based例如：頻繁訪問(wèn)本站的前N名的用戶地址2)path-based例如：被頻繁訪問(wèn)的前N名的路徑3)group-based例如：被頻繁訪問(wèn)的前N名的pagegroup2023/9/1WebUsageMining47WebLogMiner[14,15]經(jīng)過(guò)清理的log數(shù)據(jù)以DataCube的形式存儲(chǔ)在Database中提供OLAP提供包括序列模式和關(guān)聯(lián)規(guī)則在內(nèi)的多種數(shù)據(jù)挖掘的方法SimonFraserUniversity

2023/9/1WebUsageMining48WebUtilizationMiner完整的logpreparation,querying和visualization主要提供sequencepattern的發(fā)現(xiàn)提供MINT查詢語(yǔ)言3)提供樹(shù)結(jié)構(gòu)的sequencepattern顯示方式http://wum.wiwi.hu-berlin.de/德國(guó)柏林Humboldt大學(xué)商學(xué)院2023/9/1WebUsageMining492.Trends2023/9/1WebUsageMining50PAKDD20011.ServerLog提供的可用信息太少2.動(dòng)態(tài)頁(yè)面的大量使用使得分析log更為困難3.Session的分析一直是個(gè)難點(diǎn)[19]4.其他一些數(shù)據(jù)沒(méi)有記錄：例如：Searchkeywords5.Crawlers的過(guò)濾6.巨大的數(shù)據(jù)量及其自動(dòng)轉(zhuǎn)換7.MarketLevelinsightRonKohavi:

MiningE-CommerceData:TheGood,theBad,andtheUgly.*WUM在E-Commerce領(lǐng)域內(nèi)的應(yīng)用還剛起步2023/9/1WebUsageMining51WEBKDD’2000主題：WebMiningforE-Commerce--ChallengesandOpportunities2023/9/1WebUsageMining52[16]提出了Recommender系統(tǒng)的在E-Commerce領(lǐng)域內(nèi)的發(fā)展機(jī)遇和挑戰(zhàn)：1）結(jié)合多種用戶數(shù)據(jù)的,建立subtlerecommender系統(tǒng)*DemographicInformation例如：registerdata*PurchaseData*ExplictRating例如：用戶對(duì)產(chǎn)品的評(píng)價(jià)*OwnershipData2)Recommender系統(tǒng)和市場(chǎng)分析決策結(jié)合(*)3）用戶數(shù)據(jù)的共享4)道德問(wèn)題5）擴(kuò)展性*Recommender/Personalization系統(tǒng)可能的切入點(diǎn)2023/9/1WebUsageMining53WUM技術(shù)與現(xiàn)有的E-Commerce系統(tǒng)集成可能的切入點(diǎn)2023/9/1WebUsageMining54改造和構(gòu)造新的算法，把原有舊的數(shù)據(jù)源（例如WebLog）和新的數(shù)據(jù)(BusinessData)結(jié)合起來(lái)，用以發(fā)現(xiàn)新的UsagePattern可能的切入點(diǎn)2023/9/1WebUsageMining55在文[20][21]中把把用戶的TranversalPattern和purchasingPattern結(jié)合起來(lái)考慮，并構(gòu)建了一個(gè)算法用以挖掘用戶的largetransactionpattern(使用規(guī)則來(lái)表示).可能的切入點(diǎn)2023/9/1WebUsageMining56可能的切入點(diǎn):提供完整的應(yīng)用功能1.數(shù)據(jù)收集，清理和轉(zhuǎn)換2.數(shù)據(jù)存儲(chǔ)3.數(shù)據(jù)挖掘4.個(gè)人定制5.市場(chǎng)分析和決策2023/9/1WebUsageMining57可能的切入點(diǎn):提供完整的應(yīng)用功能2023/9/1WebUsageMining58Conclusion[1]JaideepSrivastava,RobertCooley,MukundDeshpande,Pang-NingTan,WebUsageMining:DiscoveryandApplicationsofUsagePatternsfromWebData(2000).SIGKDDExplorations,Vol.1,Issue2,2000.[11]RobertCooley,BamshadMobasher,andJaideepSrivastava,DataPreparationforMiningWorldWideWebBrowsingPatterns(1999),KnowledgeandInformationSystemsV1(1).2023/9/1WebUsageMining59[1]JaideepSrivastava,RobertCooley,MukundDeshpande,Pang-NingTan,WebUsageMining:DiscoveryandApplicationsofUsagePatternsfromWebData(2000).SIGKDDExplorations,Vol.1,Issue2,2000.[2]RobertCooley,BamshadMobasher,andJaideepSrivastava,WebMining:InformationandPatternDiscoveryontheWorldWideWeb(ASurveyPaper)(1997),inProceedingsofthe9thIEEEInternationalConferenceonToolswithArtificialIntelligence(ICTAI'97),November1997.[3]WWW.W3C.ORG,W3CWorkingDraftWD-logfile-960323[4]WWW.W3C.ORG,WebCharacterizationTerminology&DefinitionsSheet,W3CWorkingDraft24-May-1999[5]E.Cohen,B.Krishnamurthy,andJ.Rexford.Improvingend-to-endperformanceofthewebusingservervolumesandproxyfilters.InProc.ACMSIGCOMM,pages241-253,1998.[6]T.FawcettandF.Provost.Activitymonitoring:Noticinginterestingchangesinbehavior.InFifthACMSIGKDDInternationalConferenceonKnowledgeDiscoveryandDataMining,pages53-62,SanDiego,CA,1999.ACM.[7]MikePerkowitz,OrenEtzioni:TowardsadaptiveWebsites:Conceptualframeworkandcasestudy.ArtificialIntelligence118(1-2):245-275(2000)[8]MikePerkowitz,OrenEtzioni:AdaptiveWebSites:anAIChallenge.IJCAI(1)1997:16-23[9]AlexBuchnerandMauriceDMulvenna.Discoveringinternetmarketingintelligencethroughonlineanalyticalwebusagemining.SIGMODRecord,27(4):54-61,1998.[10]J.BenSchafer,JosephA.Konstan,JohnRiedi,Recommendersystemsine-commerce,ProceedingsoftheFirstACMConferenceonElectronicCommerce(EC-99),November3-5,1999,Denver,CO,USA.ACM,1999158-166[11]RobertCooley,BamshadMobasher,andJaideepSrivastava,DataPreparationforMiningWorldWideWebBrowsingPatterns(1999),KnowledgeandInformationSystemsV1(1).2023/9/1WebUsageMining60[12]J.Han,O.R.Za?ane,andY.Fu,``ResourceandKnowledgeDiscoveryinGlobalInformationSystems:AScalableMultipleLayeredDatabaseApproach'',Proc.ofaForumonResearchandTechnologyAdvancesinDigitalLibraries(ADL'95),McLean,Virginia,May1995.[13]O.ZaianeandJ.Han,``WebML:QueryingtheWorld-WideWebforResourcesandKnowledge'',Proc.(CIKM'98)Int'lWorkshoponWebInformationandDataManagement(WIDM'98),Bethesda,Maryland,Nov.1998,pp.9-12.[14]J.Pei,J.Han,B.Mortazavi-Asl,andH.Zhu``MiningAccessPatternsEfficientlyfromWebLogs'',Proc.2000Pacific-AsiaConf.onKnowledgeDiscoveryandDataMining(PAKDD'00),Kyoto,Japan,April2000.[15]O.R.Zaiane,M.Xin,J.Han,``DiscoveringWebAccessPatternsandTrendsbyApplyingOLAPandDataMiningTechnologyonWebLogs'',Proc.AdvancesinDigitalLibrariesConf.(ADL'98),SantaBarbara,CA,April1998,pp.19-29.[16]J.BenSchafer,JosephA.Konstan,JohnRiedi,Recommendersystemsine-commerce,ProceedingsoftheFirstACMConferenceonElectronicCommerce(EC-99),November3-5,1999,Denver,CO,USA