大數(shù)據(jù)基礎(chǔ)課程設(shè)計(jì)報(bào)告_第1頁(yè)
大數(shù)據(jù)基礎(chǔ)課程設(shè)計(jì)報(bào)告_第2頁(yè)
大數(shù)據(jù)基礎(chǔ)課程設(shè)計(jì)報(bào)告_第3頁(yè)
大數(shù)據(jù)基礎(chǔ)課程設(shè)計(jì)報(bào)告_第4頁(yè)
大數(shù)據(jù)基礎(chǔ)課程設(shè)計(jì)報(bào)告_第5頁(yè)
已閱讀5頁(yè),還剩25頁(yè)未讀 繼續(xù)免費(fèi)閱讀

下載本文檔

版權(quán)說明:本文檔由用戶提供并上傳,收益歸屬內(nèi)容提供方,若內(nèi)容存在侵權(quán),請(qǐng)進(jìn)行舉報(bào)或認(rèn)領(lǐng)

文檔簡(jiǎn)介

1、大數(shù)據(jù)基礎(chǔ)課程設(shè)計(jì)報(bào)告一、項(xiàng)目簡(jiǎn)介:使用hadoop中得hive、mapreduce以及HBASE寸網(wǎng)上得一個(gè)搜狗五百萬(wàn)得數(shù)進(jìn)行了一個(gè)比較實(shí)際得數(shù)據(jù)分析。搜狗五百萬(wàn)數(shù)據(jù),就是經(jīng)過處理后得搜狗搜索引擎生產(chǎn)數(shù)據(jù),具有真實(shí)性,大數(shù)據(jù)性,能夠較好得滿足分布式計(jì)算應(yīng)用開發(fā)課程設(shè)計(jì)得數(shù)據(jù)要求。搜狗數(shù)據(jù)得數(shù)據(jù)格式為:訪問時(shí)間t用戶IDt查詢?cè)~t該URL在返回結(jié)果中得排名、t用戶點(diǎn)擊得順序號(hào)、t用戶點(diǎn)擊得URL.其中,用戶ID就是根據(jù)用戶使用瀏覽器訪問搜索引擎時(shí)得C00kle信息自動(dòng)賦值,即同一次使用瀏覽器輸入得不同查詢對(duì)應(yīng)同一個(gè)用戶ID.二、操作要求1、將原始數(shù)據(jù)加載到HDFSF臺(tái).2、將原始數(shù)據(jù)中得時(shí)間字

2、段拆分并拼接,添加年、月、日、小時(shí)字段.3、將處理后得數(shù)據(jù)加載到HDFS平臺(tái)。4、以下操作分別通過MR與Hive實(shí)現(xiàn)。查詢總條數(shù)非空查詢條數(shù)無重復(fù)總條數(shù)獨(dú)立UID總數(shù)查詢頻度排名(頻度最高得前50詞)查詢次數(shù)大于2次得用戶總數(shù)查詢次數(shù)大于2次得用戶占比Rank在10以內(nèi)得點(diǎn)擊次數(shù)占比直接輸入U(xiǎn)RL查詢得比例查詢搜索過”仙劍奇?zhèn)b傳“得uid,并且次數(shù)大于35、將4每步驟生成得結(jié)果保存到HDFS中。6、將5生成得文件通過JavaAPI方式導(dǎo)入到HBase(一張表)。7、通過HBaseshell命令查詢6導(dǎo)出得結(jié)果。三、實(shí)驗(yàn)流程1 .將原始數(shù)據(jù)加載到HDFS平臺(tái)2 .將原始數(shù)據(jù)中得時(shí)間字段拆分并拼接

3、,添加年、月、日、小時(shí)字段(1)編寫1個(gè)腳本sogou1og-extend、sh,其中sogou-log-extend、sh得內(nèi)容為:#!/bin/bash#ininfile=$1#outoutfile=$2awkF't''print$0"t"substr($1,0,4)“年t"substr($1,5,2)“月t"substr($1,7,2)"日t"substr($1,8,2)"hour"'$infile>$outfile處理腳本文件:bashsogou-logextend、s

4、hsogou>500w、utf8sogou、500w>utf8、extrootonaster:*vintsogou-logextendrshrootc?naster:*bashsogou-log*extend.shsagou.5DDw.utf3sogou.5D0w,utf8.extroot(?nastern#lesssogou.5OE.utf8rext結(jié)果為:1十,2011123OG0OM15573751;依Rflwml3ftQ七%鈿1】國(guó)甘亦;F的切精1Mtp:'i.qiyi.rtw/州口工12月2日|OOtawr?CZ1125員由727324工3翻凡上修也休31http

5、:/'/BaokDetail.aspuPBn-okIDtlESflaOiULevel=17011卑13月DCfitur20113000001737我monlTS加7sL5Stf57135011rrp:/oohour?叫1129W6W6Q-cX%"Id即恥d9*宗市化大學(xué)岳宅K1http;/Iib.acnur4thi.Cn/21311年12月3E1日OChour201112SOGOOGGfifNA21n64d曲加大g371uCfiSf土堤代理21urrpV/prB2011123WDC0G9版Ngg7gl白Ma0f67ti2口肥已1勵(lì)了怫于后清1ht

6、tp:/inuie-dn-ba-.匚revizf1US5M/201Ifc12月3GEmohnjr?0ili23OTEroa(?sgsssbefr口謝xaFge加15曜g,youku11-:-r.*THiku,LE/20112月MEDOhvwr?CB1230000009599cd次好在1謝加6沖”代叱d可迸音光說5KL"1+lttp:/hf,tlQUSeK5.cai251112A3EJ3眥。ur2日11129OOOO10723Cdf7b&ES32a37cd16sb731ffi74陪產(chǎn)亮網(wǎng)枇大全1hrt|>:/hA4.k232i.criiira日n配12月如日日由our20

7、11U30000010295f887蚪dMG%15"的U找912二淮轉(zhuǎn)114Lk.*QHR3.將處理后得數(shù)據(jù)加載到HDFS平臺(tái)hadoopfs-putsogou>500w、utf8、ext/BrowseDirectory4.以下操作分別通過MR與H i v e實(shí)現(xiàn)LtiMYHtHiltiup3ktwRopiluIJnmBIex.Ii3U«Nanifl£SrWiF-tSf-biHLflMDBQoniKifpIrf-RAr-Tfrootmpv>yilcmpMb1Mbciinmj.jhirnwhfa*r-iI尸EJ3UMPi14MBpjfHjuso2Hm“q

8、I口Q。(VWiTviAiTUpdld>Ml1!QVrmpFjFWHTMpM>nrn>jnO.laMWI、hive實(shí)現(xiàn)1、查瞧數(shù)據(jù)庫(kù)2、創(chuàng)建數(shù)據(jù)庫(kù)3、使用數(shù)據(jù)庫(kù)4、查瞧所有表showdatabases;createdatabasesogou;usesogou;showtables;5、創(chuàng)建sougou表:Createtablesogou(timestring,uuidstring,namestring,num1int,num2int,urlstring)Rowf0rmatde1imitedfieldsterminatedby't'6、將本地?cái)?shù)據(jù)導(dǎo)入到Hive表

9、里:Loaddata1ocalinpath,/root/sogou、500WAutf8'intotab1esogou;7、查瞧表信息:descsogou;fkTesteri-rhjveSU4j:CLtsapmtPicontaifiaHultipleS.F4Jbindings.SUM;ovMbincinginjir:flit:/utr/lKil2hiUb/lojJj-tlfaj-mfl-l.6a2.jiri/arg/iliMjgl乃mHcLc郵*Eqd曠.CliSil5Lf4J:FkMbindinginTjar;file:/usr/lKal/ha-SoGpjstjar.haJw:.cMn

10、a-n/liii«3-Ljg-l1iL九。,jVI®g/li”5。U乳1讓皿事瞿面WdMI.IV4J;5f*http:/irw!*1f4j-Q>htwlMiultipilitf«ran.5LFJIJ:Actaidlbirdingisnttyp-eor1,apache,li.sif,*J.1054jlogeirFactnrytollingjnitialindusiT'OwifsguritK'ninjvrsfilti/nrAmlAiw/lib/hiw'roimn23.4Bar!/hiiffi-logJi-jSBprnpcrtimAppTc

11、ttnwHitsonK%0d/ecdtainHit史2and(1期BDtbeJWAiUblwintliefutureuvrsioas.Commi嗎&diflermtM95心(i.t,t«1f.vrkj"gmgMkvtLKreltAsn.tii中君由icwdjtaba&K-俄dvtjult域飛虜OKTltptaken:期晚拒:0小l)ive>蜘5cgou;TzretmtET:a.ltessones,F&tchedh1nawlshi*crantditafeite原in田taker:91m222seconds"上w*usbbuyu:然Tiv

12、etaker1機(jī)D3、K??谟蒱owtdblo%!DKTia*tkmri9EMSs旺口ndshjLwotfealftablesogcnj11u&stringruui>dmtwnistring.iuilint.M,?inL.url«tri«g)Rumfanatdtlin>itedfuldatftnii*tfedby'Xt'jOKT2Mtaer:1.21S弓口Ms江嶺ldkdmbciilipim,MmRQu,野*urtfsunat»l#Mg。比ln4mdatitotable就卯弧制的第匕飛u山1111nin.2Mlintintftr

13、ingTinetaken:0.655seccnis,Fetched:6rw(s(1)查詢總條數(shù)selectcount(*)fromsogou;iXw3-sel-ectcounnt(frovsoqouklWMJMj!R1W-WHWnSTBpnCTTHiw2Isainthefuimniera*口口.Ci>nid:e-ruiing上dijffff-rmteKtrulian(i,e-.te,iparicnrusinqHive1,MreIea5w.Queryig-Ed、押11?81»41207ggM3加瑪和目80趾W9<Totalj白M-1LAJjnchinq!Jeb1wt&

14、;f1NmibcTofreducetasks-del.e-rMj.ncd.1t匚口qj.Ietumej1Ifli9據(jù)tncZn*the«v?fd9«hr/1rM<.”(inbytmJ:uthive,bksc.Fcdij-c-Qrc,byt«.向-rDduca-r-TiLMbQcIn4rdertolisltth*nkIplmnumber中才reducer«iMthiw.«KK.F*du<*r.BlK-nunbaF¥Iniordertosetaesns-tantn<_nberofreducersMtmiprfStarti

15、ngJah二jabIMMZiIBiMTBiCEJGfllrTrackinqi>LHL=h±tp./Blaster.BCBB/pravy/app7icatiOk1X35須53祝_帥1/KillCuman/lier/locial/tijdanp/'bin/hadaapjot!tillja|s_5343i7Gra.'307Sfi_e0BJ由#5即j的3018.320?2OJS-12-02N0112耽saw-ij-qs2。上電-12-m11111r.Ear附fortage-linumber19EdS;Q3,W313l恥2.即.003m2m1342:33r371l)r&l

16、t;2i如;M5J5tag«1Stage-1n«g«1Etaa-1SIage1mapm邛leant,#rRappersi3jofrvdiKf6b如)匚u佻.Cunulatxve-1如c*g.Cunulstlvt1rMM”4i11G,Mapf%MlllE'*TttalEtffiulativnCFLriifiia:131»已會(huì)四£««£_C_-.t._31i.!,<1rcduco'GHt/OiimLilAtivqCPUll_19emreduce-lows;口"War上岸野mjI扎融

17、71;cMdpfUididL*J,b5LiIgbii11St-age-St*ge-11rap.3Reduce1*11*!:虹CE骼TotalMaffiBducciCFUTim學(xué)st:!-ZiwulativeCFU_13.02&ecKIFSRfdd.5737OZTI37FEIFS13cocandiENObkocbut"QCX2second?tfetchedi1eWs>H(2)非空查詢條數(shù)selecnametcount(*)fromsogouwherenameisnotnu1land='';itcount(fromgdoliwhorenabienotnull

18、an比nanoions.Considerusinga<differentexecutionengme41revtez«sparXIarusmg1Xnel疝msQU喏中10-2打,。1日12。工13元海丁必獷及7-©州7和8-融融-盯箕與陌歌了93Totaljobs-1IaurichiirifJob1o>u<tof1iMjbrofreduce-tasksJet*minedJitcompiletue-iL工作ud屯rtochangetM*電rugladfr&-redweer411nl匕享1電小,:»eth±ue_me-e.redun

19、ce-rs.-hytc,pe-r.redure-r-<numher=-tnwrflertollnitthefi-ygu*r)5Ueetfu«CEF1EAthlVOBB3WE.rdUIEi»rn,fHaK"CF|MhofInordertosetaconstantnLnber口卡educersi司做T*ob,一值!(g*-«numbgeStartinqJeb-job_lEt4-3B7QSSD7B6_OOD3,TraukirqIJR_-h'tt'p=/astqr:BOSBrprcxy/app1zca±ioni_LM3S7

20、7;50?0&_c6ai/CillC(wnar-id=/usrToczhiadaap/biri/hB)b-kill&b1M357EJS507a&EiOG3叩33010-12-02201J&-12-fi2201911-0320IB-12-02201012D2jaia2U1B-12-02iriitcbHnai-iQnf-flr力5白叁St4qs-15t*ge-1.則61g七.qu-j.5t4QB-)6Tng。-itdge-1,P-ag,p=111%.13P,Eg.lap(BXftj11dp*了立.np-1BO1.lap1匕6otmd-ppan±力聞力口af

21、Edu-greduce=g.roduco,OV.reduce-reduceOV<ZgulCumuilatCumuIEk*ti.weCumuitaTiwrwlu<arc=1r>odL|C:0CRl.ivaCPU.Q,DIeocJediceL-dnuldtlveLFUIB.4bsec.*tAr*dhK>6Cunul4tCPUtin到j(luò)10牽穹i:屯Hd&附信亨守EndadJab-jcsb_Hmm:7日S3日TB巨_日EJBm電口匚孑師£Meh5te-Stag«-T;Hap;3FteduCB:1CLnulatimCPU;IB.415secHOFSi

22、Fteedi;57JTO4.344HDF5Wri.tiK:10/勺UC注,Tfiialdu亡合CiPUT:B岫件Jp后e<i1G號(hào)白金由門通與。no&na(3)無重復(fù)總條數(shù)selectcount(*)rrom(selectfromsogougroupbytimenuml,num2,uuid,name,urlhavingcount(*)=1)a;Tirr-o(4)獨(dú)立UID總數(shù)s e 1 e ct cou n t( d is t i nct u ui d) f r om s og o u;(5)查詢頻度排名(頻度最高得前50詞)sei e ct name,count( * ) a

23、s pd from sogou gr o up by name o rder by p d d e s c li mit 50;imf am-atin for Staqc- Lj nunb-erK. , UR jl i n I?M+.HMueCW cut* : 藥豌。HDFS Bead; SZa-F-OJ+IZ hDFSCPUCFUCHJCFU CPUr«du<e roduiZQTntbI亡 CPU Tu-己 SpEL1J&2 664UtQ3:M.7D9 14:04-J7.640 日修一心14ie4a?lBBU】切£日鼻;:3日.三工】 4g04=34B44b

24、 14iO4 37a603 14:£I43Q-CQbT3g« 1 St看產(chǎn)1 Staqe -15Tig«a 1 Srag-l 腦“咆1 寫作nq« -1 5Hlre -1 萼*is/ r i 二產(chǎn)-1ir-edurcer i 117.3 日9E.Q623.2720.1CIJR_ - http ::日日BB/proicyfsippLixialzud-hill J-5amg5CPSe/HMiHadonp jmb jDia. im-mI?- /2S1.0-JL2 -H2 3HB-13-U 201S-12-02 2Qia 12 C2 2B1B-12-97 2&#

25、187;1« 12-a22BLB- 12-8OjiulaiLLtfa CPU; 3sjftJ sec 小»2O.Erodite - 1.iHivldtiwe CfJ JO-3-*rwhj仁號(hào).C jHulatiue CPU 93 笛 «bcroducQ = 1 臼BL,匚uLnftug CPU 33 .secof -iiappera i 31 p nuhber nf EdU - grHiiK/= 6.cwiiiaitiw redttce - EbLuh jlalive reduce * 6* 匚LMulNig rau::e = g. Cur jTat- 0.5即=

26、 11%. map - M3$ M祁樂丁、 國(guó)=76%.,. Ug, ri” - liBK, (3 - lOtTh, mop - 1OM. m* - ion.Ended- Jd& - iob_ 154357G55O796_0OG*HjpfMiiE.0 Lynched r5E向口白 &1:辦口曰-1;劉的;3_RaliJKg: 1_Starting 1口白 - jeb_lS43S7IK&C7Be_DCES. Tracking UI ( Oirwih4/ u1 f /1J i i/h«iriagp 4b,hi-tf'-ai- cal actLrKt uid

27、.d J f rfn aagoij;理*WNk明0r 士 電dMT oe 才殖 11/Me 1G The tutute vefLQutrw I口- raat 7ClH17Oj!14Cli4fi ZBdAdBe- 9433-43113-377c - fIS37 3E4ebBBTdIdI )dI>Gi - 1 "LIWflChlRi MO I t<it M 1Num1!er of reduce tasks dfftem±n«i at ccmpile1 tuie-j 1Into> 1心口產(chǎn) the g叼& l-o*d or ;TUc./丁 in

28、 byt*b 115-et tilwe .貯xec setlwEPB bytes per reduc sr-unber-In -nfdrf td IUhit the rr-auriiLn hUrtibci- ftf feduieerizs«t ti iye r bhoc r F4du-c fi r& .mair-nuHb« r>In nrfler to set « ccNntJiHit nwitwr q十 retfuicers- jib - r®i«kiG:E="Anu«bi»c1 際,Cu*ulau

29、vp> CRl JL12 總 44KHadDDp 0!> infonnation for Stag?-2 . rairnber 2DI11-L7 fl> 13鳴電:£rag« 3 巾邛-m田-口 3:Eife=$7.517 STaqa-J map - 1e0%, IL gr附目如 5tdge-2 map = im.皿 CPU CPU CPU CRJ CPUr«duc# 不 BeJIuce - F-fidilCtf radnee - rtduE* reduce -FM10 -5Tafl«p-l - 15t»qc- 1Stage

30、-1 w1 Stage-1 告Ugi JHDFS Ftead= 573717EE1 HDFS kl,t«= 3sl QJCCESE山$ -Ei S7T1 *rf$ riLt釗 ICT SUCX£$S13:5,713,9*4 13gMLG1 Ui»ilV.2O L3139t22.34£ Uli»i».JlM I3;5«2fi.631 13M9”"口Ojpuli»t LV¥Eknuldit lira Culmg OJilatiwn占ppoEE lr niaxtier f reduce-rs _ 1A

31、k二 221.S72 s-iMDndE r Fatcho-S: J rw I s IIX a IBQ.gi 1M.«7 111.01291®-12-W MW-12 取 2B1B- LZ -ELZ Wil-S 20113-12 02 熱】L2-M HtpflBAiE'ie- 9 fend-sd JobI IU4C hiing hiirdbar 口Tot!iv? m_i Iir* i I Minwtei '<? tn,d Q&0 riec:-lf!b_li433 T3557BC_0a4Jw 7wr af 3riadijc*dfit«fi&

32、#187;inwd a t :cap Lia t wa= 1PdlK. - gnaduu。- Eu»dLat$CPU ,xretfuce = Cwiolatlv? < 2 3 secMl|flvilKgh Ttt-il c umI at jw11Ji.il 2 VWVh4v 9W1 hvkLnd-sd Jeb - Ipb_i54357U557&e'_06C5HdipRfiftlCC JefaE L4Ui|TKh«l IStAja-EtafA -1 : Rap - 33 CLflulAtXu CPU: 11” BS aac0|E >13詢 F Ei

33、r 1 F*此EC0nm,運(yùn) k;產(chǎn)eoral Rapfwdi-e CPU Tuh&1 mIrujtm 55 -secon®In *工工M忡 Th?10iiI1 f®r d "lk4(in-R«+ hiv*. Ek" rMilCffrg bytes per rpfiuEer-<nufibera-tn trfer 1-$ Limit Lie -f>BJurfiu« Hurhbvf 9>f Mu<七才l s«t hiv* f 9X9C. rodic art .niiinijnbs r>In

34、!&> t# fiat a c«ni tant nuii>«r a t f4du>tfif£ Hip咐dbL中,j口& .Xu£百mf 0 1rAStartififl Mb - j*t_15*35?fl5»3SLlE»C5. lr*Ckl<ig URL - h-t tp /?r i SW&I p roI IC 9t 1M_1W957IM5OTB6_K®S/Kill frtHF.pd « 修尸八稗色整 job kill j通中那=ietw. 士” - INJWi m”

35、- i n. map IBei. MUp * IDCMb. map - ieo%r mjp - iM)W.iLfe-gnwicid印tk若nljjihub?mtisyi&tkraildftleji收fsCurE-索Tw-口式患川口柒,/工1川硝QB-miTiwyofTatildri?LJjjndiuii)a*mt;a1?2Umber曲心也iMhimt匕陋Hked.叫Mfr»M*TUliiL»!Jnpbf沔ra整1tH弱fpnar54ire-r|Rbr«l-seihL&i®.i£ui.er91jta.wr.Inedcr2Iuiei

36、htki)hi«inMtiilrZ&r:khr.*="J.二門¥個(gè)n(ib=jHir>jiaiderEautaNitwitimertTraducentIfT勺*憤才.橫,114和案獨(dú)沖時(shí)rtuq劇j:£l%35花藥?36二總工T依tinpUL.11。:3,拄(由湖丁也,那1g二刈二5£手lilltwandm1Hl(Lui陽(yáng)dnzJin,hadiwpjaELi:ldi_L54S?E5flnuNb»r2£通觀位注M整電MBi?0重出12任五心心Mt“軸其出相"a?國(guó)S3珊率ITe二二二二招品17牧姓冊(cè)42

37、位:4.3招班StjgE-1也抑煙/的肅膽加曲.貼5U|t1】明卯;鋁,錨5ta|s:猶鯽Bl心g瓠“加|BIUbrlLW5t3|B-l14:07tia.lMuge1U£】卻聊:Qi僵就MB】鼻陽(yáng):如田£訓(xùn)31止蚊免N舊型*-0T:3Jfg*6斑時(shí),】14力,33.至7%正-1UfI*ifl,ndbce-AMUCt-0kFMKh鑫iMure-瓊吼二MMo瓢fkp;新,0p-Amreliter*tBrwijr#rEdUCETsdan聰g7secd吼I璃gU偽上CwlatiiN*:,:匕C»lahiOirlatzLCMilatlrtCfluliriNmur?j/Eia

38、.匏痔gim蘇卻EJ西而共<的&.JAMgEMHCduLLlrt-再22.1:密三空可特法it*_*裝飛兩*+星叩魅里西萬(wàn)黃阿如和蛀魯號(hào),哩2辛際,;亙“工比.二武三.LH-II土¥|.七比,7.二I一.4&-4c*a-trf78923phJKI三口?du?4232051茹皿回王3妝WH3.7MX欽制MnllL2u*二Ja3219JJ3232市心丁k72摘耳后電;也如此蠅駁理N鴻:20如甌】q$叫,1司R整&W糊:心,施笈91型1*12.出斷甑SBM,CuauLjEl-dD曲45fcsOjiulrn'仔EJA,CuiuljrtiYE-CPU那Mu

39、lm"CRJEi修心3CRICT.Cmiuijtr-e次孫通£JjGUKHL*。若IKP.C/矍立值UaftadirTTani.txpijU-urM5A-"M,W.IM.5UL1一風(fēng)亂XeAIMguMtl爭(zhēng)產(chǎn)落?755日11干2SQ612J%2&H2畀場(chǎng)證1號(hào)瑪大金2W叫M思擊MMTim電tkflfi;1L8.P?4&多加中nfite.fetched;50即"宙;hiv?t>I(6)查詢次數(shù)大于2次得用戶總數(shù)selectcount(a、uuid)from(selectuuid,count(*)ascntfromsogougroupb

40、yuuidhavingcnt>2)a;Fjh112SALvIjffULJail:7BiwfklImp-iFWh.jiJi”hI-s.1ml£hbC"|i*cr£10!tranhi*11.i,而/Lit喇,#酊OfM11rpHi-H_3:口加比A/RS日在皿g也二皿XiRki網(wǎng).7JfWIlM;«I.«Tfl2WC-F5?Wl:M*T*t.*GM.制*WFEM膈M£l:3!*v"ftIW防f13F»rTijcr4inm*ThjwKr*4xe*B鄴.與*f聞:1141nliI8liMlEiIjlpbiMnI1if

41、fllWKl-htnm«acuIle門juwMcch;HEsetazarlsL31riF-EfticrMi#T.*31脂毗XartaniJM-二J5EEE7al.0lrx±uR不1Hsm"L中*|ridJU-Q-|j<LiUi-*idWtIMNw沸,UliJA_2&L£鵬KNjHN外勺歲-1<W(tfMunr3-Q部/Wi«T:H:口:玨汪IEbM1n-M.Fkvnlr3rcertohintthenmunirrjitercfneCJteT:三鈍hi;e,e»ec.re:jcei3.ras=<njizer>

42、;lrorterto?et3(onstatnuberofreduce:$鈍redjce.ju:.reduce5=*njibepStartingJob-嚴(yán)即79550廢學(xué)值TryingIR=http:”密tKillQvand怔山山附渝加:in/kdo叩jobkilljib15?福眥FjetuilowtionfarSiacs-2:rwiberifir.apcers:1;ijnber2B1B-U-8214:11:001Stagenap-h.reduce=A»lfiUQ此UMSffiStaged帕口-Iffi,reduce-ft.Qimbt29M-D-B2M!14:5L841St-2唧7崛,

43、reduce=喘.CinulE酗gTotalCHuljtlveCPUtue:1seconds瑚口婀12如口耳四54Cl*聯(lián)凝U廿HJ9-!依的6明,梅汨fM/T%M:|1.47,T4iStipel.C民,山U.lMZdHlMiAM西I*A,fW!?,.-的(JLEd3ll匿MEV.HPKEdlKl陋gDJMC311Ks國(guó)g乩口WOfllldd*CM強(qiáng):TE11心1*A"WCEmiedJot=而.厚為臉)用6/三也府:就uce,狀Laurchd:Sbge-Stage1:聊3heduce;3QnulatiteCPU:i?.57secm底:電H:生.由Et;liaf,left隹仙RaX船,

44、網(wǎng)畫XI«4"MmLL風(fēng)HEXU-1第Ll;Lt-HiUiStaje4y=腦MM.境界SEWLtUrXA.MillU,T!7EteI-m.均S床TJ9ft.ULUlEHIE2.E»4皿/孫£iHEMMlitVUMlKujrt»RJVF.?rIK匕iffIKHil口士比1:奧中:LabLH;vt乳I.熱»t制工.而平耀KM耶物:I畫ike;1QrjiatiweCPU:L8EsetHffWMXIFE'F1WnTHi.<?SIMA*kRt幾酩叫whw545553hive*ItRJTuneSpent:59secondsOnsec

45、Jms«CMds,Fetcbed:1rw|s)(7)查詢次數(shù)大于2次得用戶占比selectcount(*)from(selectuuid,count(*)ascntfromsogougroupbyuuidhavingcnt>2)a;1raterIfliililLreLn.!&崖3七九加3"”工悵坨皿站IflilUilLrtOVI.SW呼:不幻喻nrWgmtt:inSU陽(yáng)£IrM油-j電厚硒5即賽耳MJiteLawhpt招蟀:虹M-2I安安雪47nri鈿陽(yáng)i-12L解后LEWmTse亨也:油sscarfi詡竹旦MHUblMa,13L序然值FeUrf:1

46、irM!Ifi俗/:必”尸情福的,工1上曲1兄MmflMMUgnwjlZ卅詢CK>£U/可】口-舊IJLlflU心口上小印叮皿亡世X山4.334小日匕Wlj£*t2Litiriiniib1u:游了WK="/rM*t*5fcEtwi+jri-E$tDH:Efp匚wirtdila5H葬手in«rt«rtocMijttinMnpUadhriyits.h泗E胖.6曲.曲On:M招.隨MilgTfindbtaUfH&Mnthi.mcjHkKOTNud冷二riariffU到tiMHLaitnib4<atr«j:en-處,第TM

47、UC:Et.MtGrtfCg樂nry對(duì)*jitim造嶼泣惻:,雪通力"強(qiáng)七9»同川用廿.用1。竽1皿1(ill:4MW11"/jr*l'KiljfciEWP!1l4k»ijtfejnleririuvihr空空:2眠7何此餐9蟆5U|t<IIFF-.也U模附煙即9口:2上串MupMfB二曳mibe-1a1(djctra:3Krt理痂避_寫軸詡曲口D,Irad則次::力耗電'2日降I?也2ia-ti-n1EU必匐釐,期0”圖*eJBIS-12-©tan-6褪142,671修,W加14:19:El.Qj岫埼,2I*舊修的14:1

48、9:14,11=liJ:11.Stafe-Li5t4F-LI引鼻西11stafe-Li5H|NS點(diǎn)Ji士固N(yùn)I心心LlipgB.p-gP*坳,5a.p=通p*3p-讖.p-».p:EnKKtVArKl-9Lmtkf-款ftfuct處全由CW=ftnriUGfBNCuBilriwOjBilatLirUihuHimlnidEwliwme,%¥覽ffU17.3!kwne瞅m3理grj招,e愉厘總,兒Cwiiniiii皿a,iw一血坐-7vle11山"值CP.M.ras-irt.nJrtrrfC*i1L1站ipW,Eka-m.CMilidw0u映n理Killgm>i

49、野IkW磁桃:Mu曲:仲“iUj”HE5做叩:jirtomtionfir,叫w網(wǎng)of那曲巧;1;而¥of3M位出曲風(fēng)亞St*岬=汽reduce,«2013-U-ffi1,:題,而蜂q岬1HL向收3=的,Guhtiv那山4曲旅11艇S:ai-2腳=1瞰閶聯(lián)=Gnuht由闡iceTflalovubtivEju:uif:?EK:mfc假刖&dedJob=jcbJ5rase勘用二IWMF廊隔二柚曲陽(yáng)1吐腳;3詞工比3QhUiw(HI:Ose甘,制中5詞而小腳;1鍬席;1:1MliUw(HI;LKncI丐IiihlJtr加1K雌Turarhij»|:xe乳lii?S

50、pent:5;臟加出陰噬c30.7Hseoid:UT:1;禽(8)Rank在10以內(nèi)得點(diǎn)擊次數(shù)占比ectcount(*fromsogouwherenumljvjJ.1jiblc;i.darch«mgetl»ei331B-1jr-iMuf電fg.byr©由campj.le,訓(xùn)*e3KM*WhliQyIDTotaljobLmghinqhiLnber©Tiniarder電事(Itlviniorder_15£|357»550786_Launched:Map;SHAduCE;1GifuTuw-5印mt1H孫.OfiftMd1*"三21

51、4eJ1e428£R3J丁八,11CumuliT!ri.,«kwuyrCLfl1ULdtlVQCuffluld1120ia120.2Qie-i2-a361。23加HF%2DlS-12-a衿整唐山必*FBEndedJob帆i*拿7MeStriQiT-5titar護(hù)EtAi9iHi-lrEtgp-1rStd9eIiCupiRil4tCFiJrweotnefc320m5aczdiJuu-(A.Cm9iULbtiVe&劃lbreduce-g,,和<jC“hcpuureduce-100%.LJiiuiLat-lwa_PUhiap-EAu值3口一-注2map-1口工lim

52、itthemaximurrnjirnisekofreducers:ic.p-du(行/看H4K-TumOr>Inordertosataccmtantn-jnlwrofreducers;ebIrtepreduCA.rredufi-nmbef>'StirtinflXM-0b_1543570550706_0313pTrackingURL*1Kill匚mmnnd-/uis-r/tac.al/hadoop/binJhadacpob-killinfairuticpnf&r號(hào)17中nl,nunbarofft斗&:.rodure*rcduca-Breduicc-U:/nHS

53、torjnQFin/prnication_M3.57Q55073&_O013/|_1M3B7Q5BQ7BE_EM313hjibdrofr9duc.$r&sL(9)直接輸入U(xiǎn)RL查詢得比例selectcount()fromsogouwhereur11ike'%'Leetcount("franj/guuwiierciurlLxRe11I®intH4布VWF®iflfliBConsi<lflTusingadirfer«nce>ecutichntneIriEi.tez.spdrhorusIaqKl*l.XrtflwtB

54、S,如.¥工口三8101。1純式?乂6/丁用日。5543目3筑制"足3中加”TotaliotasLAuh£hlHd|Jftb1MT*fLihmberreduceraMtadeT«riMlr>dK#rc郵»l.l>tijneeIh-isr4er弋。chanv?thehei"建療lo«4fwr«七收已Cinbv*tes11lWthIV&,exec1,F<njt|CfElbyte導(dǎo)-p?r,7w士5bh挹InordsirtolimitthnanmKimLVnunberofrsdoica-rs=:

55、MXhlwt-*k事r4Muf旨/審Q*KHimbi®r>InoFilert?>srt«匚on零tan*numberofreducersiwtmWreduce.jiobedu皂-MLfnbmt”StiFta9口必-jcitl54.3S7KK5G7SGfl<nJ.TracrtinLFL-h11p3iLtar;OOBSpraAti4nLE4357MMm_:0014/-KillCamand-/hMr/loc-aL/h«dDcvp7bi>n/had«Dpjob-kill|ob154.157&5M7860G14Hjd口pjob

56、87;1413«201fl-12-O212«2B1B-12-G23OU13.C3291fi-12-ia2U-JjLnfornutzohfar51-1±nunberIGMMJR一一一14-26-12-feWIri20-13P9m14二三IB二N日J(rèn)OGS1土乳.3%3»1*2&-26.14.2Q.2Bn4Q1電皿¥-Staoi-1m/i>=St-*!*119l<m>|s-g,11片.37VSt:aiqo-1map-E,ffiAfb«'T的*GEMe-l1O£.tn«p-LEg.ofH

57、ijipparE£3enlwborfredhice«gr&Jwce=CuPulaitierMuc?-g,riw?roducoirs:ireduEiQrMuU*-ireOtH日-redktce口BV,Cuvulativ«MriCmkjIJi'liv*CPUCPUCPUCPUIMaipiRjodurcTotalcifTiulivaCRJtifH-s:!cacondh;彳日Flhei»cEndfldJob-Je*_kMi&7C5E<37efe_£Wj!14rtipRMliiehih4lmvtihi>*d,Stage-Stage-1.Kj4>l3Reducel1CuvruLati.weCFUiJL9u3Bsec,ICTQrfTMg,Cgulatt府CPU17.57ion.uvulBtiveffU'lt.aoMEHDFSFtoadj573704-303FDF5tfrxtToralMa(i>ReduceCPUTim區(qū)的亡1#se<oM&3H66專包(10)查詢搜索過"仙劍奇?zhèn)b傳“得uid,并且次數(shù)大于3se 1 e ctuu i d,c ount(* )uu from sowherename=/仙劍奇?zhèn)b傳gro

溫馨提示

  • 1. 本站所有資源如無特殊說明,都需要本地電腦安裝OFFICE2007和PDF閱讀器。圖紙軟件為CAD,CAXA,PROE,UG,SolidWorks等.壓縮文件請(qǐng)下載最新的WinRAR軟件解壓。
  • 2. 本站的文檔不包含任何第三方提供的附件圖紙等,如果需要附件,請(qǐng)聯(lián)系上傳者。文件的所有權(quán)益歸上傳用戶所有。
  • 3. 本站RAR壓縮包中若帶圖紙,網(wǎng)頁(yè)內(nèi)容里面會(huì)有圖紙預(yù)覽,若沒有圖紙預(yù)覽就沒有圖紙。
  • 4. 未經(jīng)權(quán)益所有人同意不得將文件中的內(nèi)容挪作商業(yè)或盈利用途。
  • 5. 人人文庫(kù)網(wǎng)僅提供信息存儲(chǔ)空間,僅對(duì)用戶上傳內(nèi)容的表現(xiàn)方式做保護(hù)處理,對(duì)用戶上傳分享的文檔內(nèi)容本身不做任何修改或編輯,并不能對(duì)任何下載內(nèi)容負(fù)責(zé)。
  • 6. 下載文件中如有侵權(quán)或不適當(dāng)內(nèi)容,請(qǐng)與我們聯(lián)系,我們立即糾正。
  • 7. 本站不保證下載資源的準(zhǔn)確性、安全性和完整性, 同時(shí)也不承擔(dān)用戶因使用這些下載資源對(duì)自己和他人造成任何形式的傷害或損失。

評(píng)論

0/150

提交評(píng)論