Ubuntu1404安裝配置Hadoop260集群

Ubuntu14.04安裝配置Hadoop2.6.0（完全分布式）與

wordcount實(shí)例運(yùn)行日期：2016-06-12 作者：朝陽沐歌我的環(huán)境是：Ubuntu14.04+Hadoop2.6.0+JDK1.8.077官網(wǎng)2.6.0的安裝教程：/docs/r2.6.0/hadoop-project-dist/hadoop-common/SingleCluster.html為了方面配置，我在每臺機(jī)器上都使用了hadoop用戶來操作，這樣做的確夠方便。結(jié)點(diǎn)信息：（分布式集群架構(gòu)：master為主節(jié)點(diǎn)，其余為從節(jié)點(diǎn)）機(jī)器名IP作用master0NameNodeandJobTrackerslave1slave2slave3123DataNodeandTaskTrackerJDK的安裝首先Hadoop運(yùn)行需要Java的支持，所以必須在集群中所有的節(jié)點(diǎn)安裝JDK，在Ubuntu14.04下安裝jdk-8u77-linux-x64.tar.gz總的原則：將jdk-8u77-linux-x64.tar.gz壓縮包解壓至/usr/lib/jdk，設(shè)置jdk環(huán)境變量并將其修改為系統(tǒng)默認(rèn)的jdk將jdk-8u77-linux-x64.tar.gz拷貝到/usr/lib/jdk/目錄下面，這里如果沒有jdk文件夾，則創(chuàng)建該文件夾，命令：sudomkdirjdk〃創(chuàng)建文件夾jdksudocp-r~/download/jdk-8u77-linux-x64.tar.gz/usr/lib/jdk/ 〃把下載的文件拷貝到新創(chuàng)建的目錄下面sudotar-zvxfjdk-8u77-linux-x64.tar.gz〃解壓縮文件設(shè)置環(huán)境變量，用gedit打開/etc/profile文件sudogedit/etc/profile在文件的最后面增加：exportJAVA_HOME=/usr/lib/jdk/jdk1.7.0_10exportJRE_HOME=/usr/lib/jdk/jdk1.7.0_10/jreexportPATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATHexportCLASSPATH=$CLASSPATH:.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib將系統(tǒng)默認(rèn)的jdk修改過來

$sudoupdate-alternatives--install/usr/bin/javajava/usr/lib/jdk/jdk1.7.0_10/bin/java300$sudoupdate-alternatives--install/usr/bin/javacjavac/usr/lib/jdk/jdk1.7.0_10/bin/javac300$sudoupdate-alternatives--configjava$sudoupdate-alternatives--configjavac4.檢測，輸入java-versionjavaversion"1.8.0_77"Java(TM)SERuntimeEnvironment(build1.8.0_77-b03)JavaHotSpot(TM)64-BitServerVM(build25.77-b03,mixedmode)配置hosts文件修改集群中所有機(jī)器的/etc/hosts，打開該文件的命令如下:sudogedit/etc/hosts添加：masterslave1slave2slave3如圖所示：localhostsiave1mastersiave1siave2siave3localhostsiave1mastersiave1siave2siave3127-0.0-1192-168-188.10192-168-188.11192-168-188.12192-168-188.12#ThefollowinglinesaredesirableforIPv6capablehosts::1localhostip6-localhostip6~loopbackff02::1ip6-allnodesff02::2ip6-allrouters注意：這里的master、slave1、slave2等等，指的是機(jī)器的機(jī)器名（使用命令hostname可以查看本機(jī)的機(jī)器名），切記，如果不是機(jī)器名的話會出問題的，并且集群中所有結(jié)點(diǎn)的機(jī)器名都應(yīng)該不一樣。3.SSH無密碼登錄修改本機(jī)(master)和子節(jié)點(diǎn)(slaveN)機(jī)器名：打開/etc/hostname文件sudogedit/etc/hostname(修改后需重啟才能生效)修改host文件(映射各個(gè)節(jié)點(diǎn)IP)：sudogedit/etc/hosts在后面添加內(nèi)容為：masterslave1slave2slave3(注意修改為本機(jī)IP)(master、slave1、slave2分別是主節(jié)點(diǎn)和子節(jié)點(diǎn)的機(jī)器名，即hostname里的內(nèi)容)為主節(jié)點(diǎn)(master)和子節(jié)點(diǎn)(slave)分別創(chuàng)建hadoop用戶和用戶組：先創(chuàng)建hadoop用戶組：sudoaddgrouphadoop然后創(chuàng)建hadoop用戶：sudoadduser-ingrouphadoophadoop(第一個(gè)hadoop是hadoop用戶組，第二個(gè)hadoop指用戶名)給hadoop用戶賦予root用戶同樣的權(quán)限，打開/etc/sudoers文件(目的：給hadoop用戶sudo權(quán)限)sudogedit/etc/sudoers在rootALL=(ALL:ALL)ALL這一行下添加hadoopALL=(ALL:ALL)ALL本機(jī)（master）和子節(jié)點(diǎn)（slave）安裝ssh服務(wù):sudoapt-getupdatesudoapt-getupgradesudoapt-getinstallsshopenssh-server建立ssh無密碼登錄環(huán)境：進(jìn)入新建立的hadoop用戶，建議注銷當(dāng)前用戶，然后選擇hadoop用戶ssh生成密鑰有rsa和dsa兩種生成方式，默認(rèn)情況下采用rsa方式創(chuàng)建ssh-key，這里我們采用rsa方式，在終端/home/hadoop目錄下輸入：ssh-keygen-trsa-P""（有確認(rèn)信息直接回車）進(jìn)入~/.ssh/目錄下：cd/home/hadoop/.ssh將此目錄下的id_rsa.pub追加到authorized_keys授權(quán)文件中：catid_rsa.pub>>authorized_keys將master節(jié)點(diǎn)上的rsa.pub通過ssh傳到子節(jié)點(diǎn)上（目的：公用公鑰密鑰）X代表第n個(gè)結(jié)點(diǎn)scp~/.ssh/id_rsa.pubhadoop@slaveX:~/.ssh/進(jìn)入~/.ssh/目錄下，將id_rsa.pub追加到authorized_keys授權(quán)文件中cd/home/hadoop/.sshcatid_rsa.pub>>authorized_keys測試ssh互信是否建立sshhadoop@slave1（如果不需要輸入密碼就可以登錄成功則表示ssh互信已經(jīng)建立）Hadoop的安裝與配置（1）.下載解壓Hadoop穩(wěn)定版我用的是hadoop-2.6.0，下載地址：/apache/hadoop/common/stable/將下載后的Hadoop拷貝到hadoop目錄下，解壓到master服務(wù)器的/hadoop目錄下（配置好master結(jié)點(diǎn)后再將其復(fù)制到其它的服務(wù)器上，一般來說，群集中所有的hadoop都安裝在同一目錄下）：解壓命令如下：tar-xzfvhadoop-2.6.0.tar.gz（2）.配置Hadoop修改hadoop-2.6.0/etc/hadoop/hadoop-env.sh，添加JDK支持：exportJAVA_HOME=/usr/java/jdk1.8.0_77如果不知道你的JDK目錄，使用命令echo$JAVA_HOME查看。修改hadoop-2.6.0/etc/hadoop/core-site.xml注意：必須加在<configuration></configuration>節(jié)點(diǎn)內(nèi)<configuration><property><name>hadoop.tmp.dir</name><value>/home/hadoop/hadoop-2.6.0/tmp</value><description>Abaseforothertemporarydirectories.</description></property><property><name></name><value>hdfs://master:9000</value></property></configuration>修改hadoop-2.6.0/etc/hadoop/hdfs-site.xml<property><name>.dir</name><value>/home/hadoop/hadoop-2.6.0/dfs/name</value><description>PathonthelocalfilesystemwheretheNameNodestoresthenamespaceandtransactionslogspersistently.</description>〈/property〉〈property〉<name>dfs.data.dir</name><value>/home/hadoop/hadoop-2.6.0/dfs/data</value><description>CommaseparatedlistofpathsonthelocalfilesystemofaDataNodewhereitshouldstoreitsblocks.</description></property><property><name>dfs.replication</name><value>1</value></property>修改hadoop-2.6.0/etc/hadoop/mapred-site.xml<property><name>mapred.job.tracker</name><value>master:9001</value><description>HostorIPandportofJobTracker.</description></property>修改hadoop-2.6.0/etc/hadoop/masters列出所有的master節(jié)點(diǎn)：master修改hadoop-2.6.0/etc/hadoop/slaves這個(gè)是所有datanode的機(jī)器，例如：slavelslave2slave3slave4將master結(jié)點(diǎn)上配置好的hadoop文件夾拷貝到所有的slave結(jié)點(diǎn)上以slavel為例：命令如下:scp-r~/hadoop-2.6.0hadoop@slave1:~/安裝完成后，我們要格式化HDFS然后啟動集群所有節(jié)點(diǎn)。啟動Hadoop格式化HDFS文件系統(tǒng)的namenode（這里要進(jìn)入hadoop-2.6.0目錄來格式化好些）：cdhadoop-2.6.0//進(jìn)入hadoop-2.6.0目錄bin/hdfsnamenode-format〃格式化啟動Hadoop集群啟動hdrs命令如下：sbin/start-dfs.sh//開啟進(jìn)程成功的話輸入jps會出現(xiàn)如下界面：hadoopfiimsteT:$jps3331Jps2948Resourcelanager2567NhiuhNMw2793SecondaryMomNodeter:$猝 ?h2：AES-2S6-CTR補(bǔ)充，關(guān)閉hdfs集群的命令如下:sbin/stop-dfs.sh我們也可以通過網(wǎng)頁來看是否正常安裝與配置，地址如下：http://master:50070/HadMpOfErvier StartupPr<hpre-E5UtElitiesDatanodeInformationInoperationLacI ArLlLeiStata DPSIfb^i啾白皿iiiiAKEll?&k.LE-lo^-kpi?lucaJfkilad Ii^xeiob.DecomissioningUa^firR?plic?t?d[B-Ucc-kc￥aJiaLast UhJm*rnplic>BtaJblaehT llcmkcTithnolir*rerplic^B Tnfi]?evnitirccn-a-tnati#!!dodiKfp,2014.實(shí)例運(yùn)行（運(yùn)行wordcount程序）1.創(chuàng)建input目錄在Hadoop-2.6.0目錄下創(chuàng)建input目錄命令如下：mkdirinputhadoopimaster：/opt/program/hadoop/ha[ioop-2.6.0$mkdirinputhadoopSmaster:/opt/program/hadoop/hatioop-2.6-0$IsincludelibLICENSE,txtNOTICE,tztREADIE,txthadoopimasterTTopt/prograiii/hadoop/hatloop-z!.&.0$在input創(chuàng)建fl、f2并寫內(nèi)容命令如下：catinput/f1Helloworld byejjcatinput/f2Helloworld byejj或者手動創(chuàng)建文本文件，并在里面放下英文文章hadoopSmaster:/opt/program/hadoop/hadoop-S.6?0/input$Isfilelfile2在hdfs創(chuàng)建/porrylee/input目錄命令如下：bin/hadoopfs-mkdir/porryleebin/hadoopfs-mkdir/porrylee/input將fl、f2文件copy到hdfs/porrylee/input目錄命令如下：bin/hadoopfs-putinput//porrylee查看hdfs上是否有f1、f2文件命令如下：bin/hadoopfs-ls/porrylee/input/執(zhí)行wordcount程序命令如下：bin/hadoopjarshare/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jarwordcount/porrylee/input//output/wordcount3查看執(zhí)行結(jié)果命令如下：hadoop@master:~/hadoop-2.6.0$bin/hdfsdfs-cat/output/wordcount3/*執(zhí)行后，可以看到統(tǒng)計(jì)結(jié)果附錄（核心代碼）packagecom.felix;importjava.io.IOException;importjava.util.Iterator;importjava.util.StringTokenizer;importorg.apache.hadoop.fs.Path;importorg.apache.hadoop.io.IntWritable;importorg.apache.hadoop.io.LongWritable;importorg.apache.hadoop.io.Text;importorg.apache.hadoop.mapred.FileInputFormat;importorg.apache.hadoop.mapred.FileOutputFormat;importorg.apache.hadoop.mapred.JobClient;importorg.apache.hadoop.mapred.JobConf;importorg.apache.hadoop.mapred.MapReduceBase;importorg.apache.hadoop.mapred.Mapper;importorg.apache.hadoop.mapred.OutputCollector;importorg.apache.hadoop.mapred.Reducer;importorg.apache.hadoop.mapred.Reporter;importorg.apache.hadoop.mapred.TextInputFormat;importorg.apache.hadoop.mapred.TextOutputFormat;/***描述：WordCountexplainsbyFelix@authorHadoopDevGroup*/publicclassWordCount(/**MapReduceBase類：實(shí)現(xiàn)了Mapper和Reducer接口的基類(其中的方法只是實(shí)現(xiàn)接口，而未作任何事情)Mapper接口：WritableComparable接口：實(shí)現(xiàn)WritableComparable的類可以相互比較。所有被用作key的類應(yīng)該實(shí)現(xiàn)此接口。Reporter則可用于報(bào)告整個(gè)應(yīng)用的運(yùn)行進(jìn)度，本例中未使用。**/publicstaticclassMapextendsMapReduceBaseimplementsMapper<LongWritable,Text,Text,IntWritable>(/**LongWritable,IntWritable,Text均是Hadoop中實(shí)現(xiàn)的用于封裝Java數(shù)據(jù)類型的類，這些類實(shí)現(xiàn)了WritableComparable接口，*都能夠被串行化從而便于在分布式環(huán)境中進(jìn)行數(shù)據(jù)交換，你可以將它們分別視為long,int,String的替代品。*/privatefinalstaticIntWritableone=newIntWritable(1);privateTextword=newText();/**Mapper接口中的map方法：voidmap(K1key,V1value,OutputCollector<K2,V2>output,Reporterreporter)*映射一個(gè)單個(gè)的輸入k/v對到一個(gè)中間的k/v對*輸出對不需要和輸入對是相同的類型，輸入對可以映射到0個(gè)或多個(gè)輸出對。OutputCollector接口：收集Mapper和Reducer輸出的<k,v>對。OutputCollector接口的collect(k,v)方法：增加一個(gè)(k,v)對到output*/publicvoidmap(LongWritablekey,Textvalue,OutputCollector<Text,IntWritable>output,Reporterreporter)throwsIOException(Stringline=value.toString();StringTokenizertokenizer=newStringTokenizer(line);while(tokenizer.hasMoreTokens())(word.set(tokenizer.nextToken());output.collect(word,one);}}}publicstaticclassReduceextendsMapReduceBaseimplementsReducer<Text,IntWritable,Text,IntWritable>(publicvoidreduce(Textkey,Iterator<IntWritable>values,OutputCollector<Text,IntWritable>output,Reporterreporter)throwsIOException(intsum=0;while(values.hasNext())(sum+=values.next().get();}output.collect(key,newIntWritable(sum));}}publicstatic