云計(jì)算-cloud l主要需求和目標(biāo)

副研究員博士 主要需求和目標(biāo)實(shí)現(xiàn)為程序員隱藏系統(tǒng)層細(xì)需要考慮的細(xì)節(jié)技術(shù)問題如何管理 數(shù)據(jù)？如何劃分?jǐn)?shù)據(jù)如何調(diào)度計(jì)算任務(wù)并分配map和reduce節(jié)點(diǎn)如果節(jié)點(diǎn)間需要共享或交換數(shù)據(jù)如何如何掌控節(jié)點(diǎn)的執(zhí)行完成情況？如何收集中間和最終的結(jié)果節(jié)點(diǎn)失效如何處理？如何恢復(fù)數(shù)節(jié)點(diǎn)擴(kuò)充后如何保證原有程序仍能正常運(yùn) 系統(tǒng)性能提升 TextText1:theweatheris Text2:todayisText3:goodweatheris Text4:todayhasgoodString[]text=args[0];HashTableht=newHashTable();for(i=0;i<3;++i){StringTokenizerst=newStringTokenizer(text[i]);while(st.hasMoreTokens()){Stringword=if(!ht.containsKey(word)){ht.put(word,newInteger(1));}else{intwc((Integer)ht.get(word)).intValue()+1;//計(jì)數(shù)加1ht.put(word,new}}}for(Iteratoritr=ht.KeySet().iterator();itr.hasNext();{Stringword=System.out.print(word+“:”+ }輸出：good: has:1;is: the: today: weather:

鍵值

鍵值

海量數(shù)

鍵值

初始鍵值

Barrier：AggregationBarrier：Aggregationand

中間結(jié)

計(jì)算結(jié)MapReducemap:(k1;v1)→[(k2;reduce:(k2;[v2])→[(k3;)][最終的某種形式的結(jié)果輸出[(k3;v3)]輸出：最終輸出結(jié)果[(k3;使用4個(gè)map節(jié)點(diǎn)map節(jié)點(diǎn)輸入：(text1theweatheris輸出：(the1weather1(is1(goodmap節(jié)點(diǎn)輸入：(text2todayismap節(jié)點(diǎn)map節(jié)點(diǎn)reduce節(jié)點(diǎn)輸入：(good,1),(good,1),(good,1),(good1),(good(is,1),(is,輸出(is,1),(is,輸出good:is:3weather:reduce節(jié)點(diǎn)輸入：(has,1),輸出：(has,1),(is,輸入：(the,1),today1),(today,(weather,1),(weather,1),(weather,輸出：(the,1),(today,2),weatherreduce節(jié)點(diǎn)輸入：(good,1),(good,1),(good,2)(good,(is,1),(is,輸出(is,1),(is,輸出good:is:3weather:reduce節(jié)點(diǎn)輸入：(has,1),輸出：(has,1),(is,輸入：(the,1),today1),(today,(weather,1),(weather,1),(weather,輸出：(the,1),(today,2),weatherHadoop:Mapper&ReducerHadoop:DriverHadoop:Run HadoopMapReducesplitsplitsplitsplit

partpartpartpartsplitsplitReducershas3primaryphases:shuffle,sortandHadoop:HowmanyTasksetuptakesawhile,soitisbestifthemapstakeatleastaminuteto用戶可以顯示配置Configuration.set(MRjobConfig.NUM_MAPSintbutitonlyprovidesahinttotheframework10TBofinputdata,blocksizeisYouwillendupwith80KHadoop:HowmanyThenumberofreducersforthejobissetbytheuserTherightnumberofreducesseemstobe0.95or1.75multipliedby(<no.of<no. umcontainersperWith0.95allofthereducescanlaunch yandstarttransferringmapoutputsthemapsWith1.75thefasternodeswillfinishtheirfirstroundofreducesandlaunchasecondwave ngamuchbetterjobofloadIncreasingthenumberofreducesincreasestheframeworkoverhead,butloadbalancingandlowersthecostofHadoop:PartitionercontrolsthepartitioningofthekeysoftheintermediateThetotalnumberofpartitionsisthesameasthenumberofreducetasksforthejob.Hencethiscontrolswhichofthemreducetaskstheintermediatekey(andhencetherecord)issenttoforreductionHashPartitioneristhedefaultHadoop:JobJobrepresentsaMapReducejobJobistheprimaryinterfaceforausertodescribeaMapReducejobtoHadoopframeworkforJobistypicallyusedtoHadoop:JobOptionally,JobisusedtospecifyotheradvancedfacetsofthejobsuchFilestobeputintheWhetherintermediateand/orjoboutputsaretobecompressed(andWhetherjobtaskscanbeexecutedinaspeculativeumnumberofattemptsperUserscanuseConfiguration.set(String,String)/Configuration.get(String)toset/getarbitraryparametersneededbyapplicationsCountersrepresentglobalCounterscouldbedefinedMapReduceInthemapand/orreducemethods,applicationscandefinearbitraryCounters(oftypeEnum)andupdatethemvia:Counters.incrCounter(Enum,Counters.incrCounter(String,String,ThesecountersarethengloballyaggregatedbytheHadoop:CurrentlyFileInputFormat,TextInputFormat,SequenceFileInputFormat,CombineFileInputFormat,CombineSequenceFileInputFormat,CombineTextInputFormat,CompositeInputFormat,FixedLengthInputFormat,SequenceFileAsBinaryInputFormat,Seq