Sparkonyarn集群搭建詳細(xì)過(guò)程

1、SparkonYarn集群搭建詳細(xì)過(guò)程由于最近學(xué)習(xí)大數(shù)據(jù)開發(fā)，spark作為分布式內(nèi)存計(jì)算框架，當(dāng)前十分火熱，因此作為首選學(xué)習(xí)技術(shù)之一。Spark官方提供了三種集群部署方案：Standalone,Mesos,Yarn其中Standalone為spark本身提供的集群模式，搭建過(guò)程可以參考官網(wǎng)，本文介紹SparkonYarn集群部署過(guò)程。使用3臺(tái)普通機(jī)器搭建Spark集群，軟件環(huán)境：Ubuntu16.04LTSUbuntu16.04LTSCentOS7Scala-2.10.6Hadoop-2.7.2spark-1.6.1-bin-hadoop2.6Java-1.8.0_77硬件環(huán)境：一個(gè)Mast

2、er節(jié)點(diǎn)Intel?Core?i5-2310CPU2.90GHz乂44G內(nèi)存300G硬盤兩個(gè)Slave節(jié)點(diǎn)Intel?Core?i3-2100CPU3.10GHz乂44G內(nèi)存500G硬盤、配置/etc/hosts及免密碼登錄本文下載安裝的軟件都放在home目錄下1.主機(jī)hosts文件配置在每臺(tái)主機(jī)上修改host文件01B.199.92.225218.199+92.22627210,199.92.222127*0*0*1fang-Lenovofang-centosfang-ubuntulfang-inacbookPro.locallocalhostloc

3、alhost#ThefollowinglinesaredesirableforIPv6capablehosts：ff00：off02:vpfi-locathostip6-loopbacktpe-localnettp6*ncastprefixlp6-allrouterssudovim/etc/hosts27fang-ubuntu1(Master)26fang-centos(Slave)25fang-Lenovo(Slave)localhostlocalhost注：若此地未配置，或者未配置正

4、確會(huì)導(dǎo)致集群?jiǎn)?dòng)不正?；蛘呤∨渲弥髉ing一下各機(jī)器名稱檢查是否生效，例如sshfangfang-centos2,配置SSH免密碼登錄如果沒有安裝ssh,需要安裝Opensshserver,命令為sudoapt-getinstallopenssh-server1）在所有機(jī)器上都生成私鑰和公鑰ssh-keygen-trsa路回車2）需要讓機(jī)器間都能相互訪問(wèn)，就把每個(gè)機(jī)子上的id_rsa,pub發(fā)給master節(jié)點(diǎn)，傳輸公鑰可以用scp來(lái)傳輸。scp/.ssh/id_rsa.pubfangfang-ubuntu1:/.ssh/id_rsa.pub.slave13）在master上，將所有公鑰加

5、到用于認(rèn)證的公鑰文件authorized_keys中cat/.ssh/id_rsa.pub*-/.ssh/authorized_keys4）將公鑰文件authorized_keys分發(fā)給每臺(tái)slavescp-/.ssh/authorized_keysfangfang-centos:/.ssh/5）在每臺(tái)機(jī)子上驗(yàn)證SSH無(wú)密碼登錄fangfang-ubuntui:-$sshfangfang-centosLastlogtn:FrtApr291S:24:OS2016fangfangcentos-$|在終端中輸入登錄命令，例如：sshfangfang-centos如果直接登錄成功而不需要登錄密碼，則表

6、示設(shè)置正確；如果登錄不成功，即仍然需要登錄密碼，則可能需要修改文件authorized_keys的權(quán)限。注：.ssh文件夾的權(quán)限必須為700,authorized_keys文件權(quán)限必須為600使用如下命令改變文件夾權(quán)限：chmod600/.ssh/authorized_keys二、安裝Java從官網(wǎng)下載最新版Java,Spark官方說(shuō)明Java只要是6以上的版本都可以,本文使用的是jdk-8u91-linux-x64.tar.gz。在下載目錄下直接解壓tar-zcvfjdk-8u91-linux-x64.tar.gz并復(fù)制文件到/usr/lib/jvm中，命令如下：sudocp-rjdk1,8

7、.0_77/usr/lib/jvm（如果沒有jvm文件夾，則手動(dòng)創(chuàng)建一個(gè)）,修改環(huán)境變量sudovim/etc/profile,添加下列內(nèi)容：exportJAVA_HOME=/usr/lib/jvm/jdk1.8,0_77exportJRE_HOME=$JAVA_HOME/jreexportPATH=$JA/A_HOME/bin:$JAVA_HOME/jre/bin:$PATHexportCLASSPATH=$CLASSPATH:.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib然后使環(huán)境變量生效，并驗(yàn)證Java是否安裝成功$source/etc/profile#生效環(huán)境變

8、量$java-version#如果打印出如下版本信息，則說(shuō)明安裝成功fangfang-centos$java-versionjavaversionM1.8.0_77lbJava(TM)SERuntimeEnvironment(build1.S.G_77-b&3)JavaHot$pot(TM)64-BttServerVM(build25-77-b031ntxedmode)fangfang-centos|三、安裝ScalaSpark官方要求Scala版本為2.10.x,注意不要下錯(cuò)版本，我這里下了2.10.6同樣我們?cè)?中解壓tar-zcvfscala-2.10.6.tar.gz再次修改環(huán)境變量s

9、udovim/etc/profile,添加以下內(nèi)容：exportSCALA_HOME=/home/fang/scala-2.10.6exportPATH=$PATH:$SCALA_HOME/bin同樣的方法使環(huán)境變量生效，而僉證scala是否安裝成功$source/etc/profile#生效環(huán)境變量$scala-version#如果打印出如下版本信息，則說(shuō)明安裝成功。fanggfang-centosscala-verstonScalacoderunnerverston2.10.6-Copyright2S62-2013,LAWP/EPFLfarigfang-centos$)四、安裝配置Hado

10、opYARN從官網(wǎng)下載hadoop2.7.2版本，在用戶根目錄解壓tar-zcvfhadoop-2.7.2.tar.gz再次修改環(huán)境變量sudovim/etc/profile,添加以下內(nèi)容：exportHADOOP_HOME=/home/fang/hadoop-2.7.2exportHADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoopexportYARN_HOME=/home/fang/hadoop-2.7.2exportYARN_CONF_DIR=$YARN_HOME/etc/hadoop同樣的方法使環(huán)境變量生效$source/etc/profile#生效環(huán)境變量注

11、：有時(shí)候修改了/etc/profile文件，執(zhí)行命令source之后還是不能達(dá)到正常的效果，則需要重新機(jī)器，看問(wèn)題是否能解決。酉己置Hadoop,cd/hadoop-2.7.2/etc/hadoo剛?cè)雋adoop酉己置目錄，需要酉己置有以下7個(gè)文件：hadoop-env.shyarn-env.sh,slaves,core-site.xml,hdfs-site.xml,maprd-site.xml,yarn-site.xml。在hadoop-env.sh中酉己置JAVA_HOME# Thejavaimplementationtouse.exportJAVA_HOME=/usr/lib/jvm/j

12、dk1.8.0_77在yarn-env.sh中酉己置JAVA_HOME# someJavaparametersexportJAVA_HOME=/usr/lib/jvm/jdk1.8.0_77在slaves中配置slave節(jié)點(diǎn)的ip或者h(yuǎn)ost,fang-centosfang-Lenovo修改core-site.xmlfs.defaultFShdfs:/fang-ubuntu1:9000/hadoop.tmp.dirfile:/home/fang/hadoop-2.7.2/tmp修改node.secondary.http-addressfang-ubun

13、tu1:9001.dirfile:/home/fang/hadoop-2.7.2/dfs/namedfs.datanode.data.dirfile:/home/fang/hadoop-2.7.2/dfs/datadfs.replication3修改yarn修改yarn-site.xmlyarn.nodemanager.aux-servicesmapreduce_shuffleyarn.nodemanager.aux-services.mapreduce.shuffle.classo

14、rg.apache.hadoop.mapred.ShuffleHandleryarn.resourcemanager.addressfang-ubuntu1:8032yarn.resourcemanager.scheduler.addressfang-ubuntu1:8030yarn.resourcemanager.resource-tracker.addressfang-ubuntu1:8035yarn.resourcemanager.admin.addressfang-ubuntu1:8033yarn.resourcemanager.webapp.addressfang-ubuntu1:8

15、088將配置好的hadoop-2.7.2文件夾分發(fā)給所有slaves節(jié)點(diǎn)scp-r/hadoop-2.6.0fangfang-centos:/啟動(dòng)Hadoop在master節(jié)點(diǎn)上執(zhí)行以下操作，就可以啟動(dòng)hadoop了。cd/hadoop-2.7.2#進(jìn)入hadoop目錄bin/hadoopnamenode-format#格式化namenode注：若格式化之后重新修改了配置文件，重新格式化之前需要?jiǎng)h除tmp,dfs,10gs文件夾。sbin/start-dfs.sh#啟動(dòng)dfssbin/start-yarn.sh#啟動(dòng)yarn驗(yàn)證Hadoop是否安裝成功，可以通過(guò)jps命令查看各個(gè)節(jié)點(diǎn)啟動(dòng)的進(jìn)程

16、是否正常。在master上應(yīng)該有以下幾個(gè)進(jìn)程：fangfang-ubuntul:jps11889NameNode11S22Master12919DataNode12264NodeManager12395Jps12155SecondaryNaneNode在每個(gè)slave上應(yīng)該有以下幾個(gè)進(jìn)程:fangfang-cento5jps16513DataNode17607Jps17656NodeManager在瀏覽器中輸入http:/fang-ubuntu1:8088,可以看到hadoop的管理界面三二|岫取Ahaurnd亡5UBM1TTE口auczm口riNiHHrnfAiirnKIP門Clu44erM

17、eme%Apps-AppflApps-即g口KtEtraMenwryVComVCctwVCqmActv七Oeoeekknic4LeIUnhutTRwidngR%Jmpteud艮仆川Lk4dT2INnlRourv4dIWkNod”hkdMModwaC-C0.口日部由口B024D3Q口口QScheddcfMBtiicsSdwutefTyptiEuduiir4T,|MlArwuiHiMbZMknMdiifYiumAloeAUXiCdipiciySchuMwMEMORYmwnfif1Q34liCatk-nnMMyAISIZvCdmcio.xMadaNodeAdd皿NodeHUP.x,4CfTiiiiw

18、i出VComVDuuVanfaxLobd：Rode，與咖，：,加好4s；LBnhearthulE.;Uq.如一;Scheduler心應(yīng)場(chǎng)RUNNING府性物麻網(wǎng)五四月及俯，Q值00685U0也了7iFKk的twWZFN整循*TgW小的如RUNNING為附物比iKieE四目粉伸則？07008GB屯fi172rackubunijlisaes3iui皎心3016陽(yáng)ahhil.RUNNING&ng.也叫星國(guó)元國(guó)同2916.411:07003BGBDH27JmckLflfKW：l?*5S.unowHlSOd3015Showingla31Hmln1_叫聚4g：:NodesoftheclustercbZia

19、.WSJ222r：gC4fdusftrr/hade&五、Spark安裝下載解壓，進(jìn)入官方下載地址下載最新版Spark。我下載的是spark-1.6.1-bin-hadoop2.6.tar.gz在/目錄下解壓，tar-zcvfspark-1.6.1-bin-hadoop2.6.tar.gz配置Sparkcdspark-1.6.1-bin-hadoop2.6/conf#入spark配置目錄cpspark-env.sh.templatespark-env.sh#從酉己置模板復(fù)制vimspark-env.sh#添加配置內(nèi)容在spark-env.s時(shí)尾添加以下內(nèi)容（這是我的配置，你可以自行修改）：exp

20、ortSPARK_HOME=/home/fang/spark-1.6.1-bin-hadoop2.6exportSCALA_HOME=/home/fang/scala-2.10.6exportJAVA_HOME=/usr/lib/jvm/jdk1.8.0_77exportHADOOP_HOME=/home/fang/hadoop-2.7.2exportPATH=$PATH:$JAA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$SCALA_HOME/binexportHADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoopexp

21、ortYARN_CONF_DIR=$YARN_HOME/etc/hadoopexportSPARK_MASTER_IP=27SPARK_LOCAL_DIRS=/home/fang/spark-1.6.1-bin-hadoop2.6SPARK_DRIVER_MEMORY=1GexportSPARK_LIBARY_PATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib:$HADOOP_HOME/lib/native注：在設(shè)置Worker進(jìn)程的CPU個(gè)數(shù)和內(nèi)存大小，要注意機(jī)器的實(shí)際硬件條件，如果配置的超過(guò)當(dāng)前Worker節(jié)點(diǎn)的硬件條件，Worker進(jìn)

22、程會(huì)啟動(dòng)失敗。vimslaves在slaves文件下填上slave主機(jī)名：slave1slave2將配置好的spark-1.6.1-bin-hadoop2.6文件夾分發(fā)給所有slaves吧scp-r/spark-1.6.1-bin-hadoop2.6fangfang-cenos:/啟動(dòng)Spark,sbin/start-all.sh驗(yàn)證Spark是否安裝成功主節(jié)點(diǎn)上啟動(dòng)了Master進(jìn)程：fangfdng-ubuntul:-/spark-l.6rl-btn-hadoop2*6/sbin$jps11522Master在slave上啟動(dòng)了Worker進(jìn)程:fanggfang-centos-$jps1

23、8630Jp18252Worker進(jìn)入Spark的Web管理頁(yè)面：http:/fang-ubuntu1:8080ORL;i危i81MK227:7077RESTURL:5(Mricii21B.I9ei.9S-Z2Tj)irnv“4AJiw新M*9E3CornInmm:BFoiaiU&ed“mgvy-wv;5.130Tq.M0Mnd由澗Mmcs:口fbjmrfl,0CanceledMen.口Runrifql-0CMHptalAdSIvIuf:，LWEWot*eraWwfeerIdAddrflK麗心CraE9222M1m2W1的鍛融13WALFWE4/1aMQ立了例QOBU*wtirlffZtllE

24、M29154739-290.199922X223Z1B.19992226323ALTYE4C0Uwd|Z4C3|DJDBUsedjRunningAppllcationis即f川aboriIDNrntMtffiOfyparNbddBuHtSidTIhhLluifSuuuriiienC所pl旭日ApfdgEi0n1口M4nwMrwtiTWUtvf口urpEn六、運(yùn)行示例本例以集群模式運(yùn)行SparkPi實(shí)例程序(deploy-mode設(shè)置為cluster)./bin/spark-submit-classorg.apache.spark.examples.SparkPi-masteryarn-depl

25、oy-modecluster-driver-memory1G-lib/spark-examples-1.6.1-hadoop2.6.0.jar40任務(wù)提交時(shí)web界面向3喈)-Clusterdu&barMobiesAten細(xì)用AmjsApps-QtjrimMarrEiizmtfedPetidingRurfirgCcmKedRunningUs忖miuLab*10-1。12GO皿3Uan5StJidul&r舊li國(guó)嚼-的血印TflSdwKLbrR&ssuahiFT團(tuán)CofMot)1ScheddaWBK3AY川zurrut一皿用口Strawra-tnirr則以位心u口ID弋rK【UPHdWcabwi述MMHlarq0(1.邛第Hyxiric史Kamp蛇SchHuErTpglfSlKMing1tn1uf1wtnssexecutor-memory1G-executor-cores1lotAllApplicationsEqmwyMwncrVCqthVCmwVCom-OKmnnjwiQnedLEUitnwftladTouiLteed.ToladFkdnNodes.Nodn24-GBQB1340J口I0DurccT,pnlAfwuiinAJkiunDTiMxomunijUocawsmemcrylOQ”.vCaresdcnernHy:31