Hadoop平臺(tái)搭建與應(yīng)用（第2版）（微課版） 課件 項(xiàng)目6 Hadoop HA集群搭建

任務(wù)6.1HadoopHA環(huán)境搭建任務(wù)實(shí)施1.配置靜態(tài)IP在宿主機(jī)中，創(chuàng)建自定義網(wǎng)橋，指定網(wǎng)段，網(wǎng)關(guān)，操作命令及結(jié)果如下。[root@CentOS~]#dockernetworkcreate--subnet=/24--gateway=HAf1bad6fa92064749b7e9df80ae3832fd0a33ba644b781c83b2d7da6510ea57232.Docker容器添加端口映射創(chuàng)建3個(gè)容器，并添加端口映射，操作命令如下。[root@CentOS~]#dockerrun-dit-hmaster1--namemaster1--network=HA--ip=0-p10022:22-p50070:50070-p8088:8088-p8042:8042-p2181:2181--privileged=truecentos:7init

[root@CentOS~]#dockerrun-dit-hslave1--nameslave1--network=HA--ip=1-p10023:22-p50071:50070-p8089:8088-p2182:2181-p8043:8042--privileged=truecentos:7init

[root@CentOS~]#dockerrun-dit-hslave2--nameslave2--network=HA--ip=2-p10024:22-p8090:8088-p2183:2181-p8044:8042--privileged=truecentos:7init3.拷貝安裝包到master1容器中，操作命令及結(jié)果如下。[root@CentOSopt]#dockercp/opt/hadoop-2.7.7.tar.gzmaster1:/optSuccessfullycopied218.7MBtomaster1:/opt[root@CentOSopt]#dockercp/opt/zookeeper-3.4.8.tar.gzmaster1:/optSuccessfullycopied22.26MBtomaster1:/opt[root@CentOSopt]#dockercp/opt/jdk-8u341-linux-x64.tar.gzmaster1:/optSuccessfullycopied148.2MBtomaster1:/opt4.進(jìn)入master1容器，安裝并開啟SSH服務(wù)，操作命令如下。[root@CentOSopt]#dockerexec-itmaster1bash[root@master1/]#yuminstall-ynet-toolsvimopenssh

openssh-clientsopenssh-serverwhich[root@master1/]#systemctlstartsshd5.解壓jdk，hadoop，zookeeper文件到指定目錄中，并重命名文件，操作命令如下。[root@master1opt]#tar-zxvfjdk-8u341-linux-x64.tar.gz-C/usr/lib/[root@master1opt]#tar-zxvfhadoop-2.7.7.tar.gz-C/usr/local/[root@master1opt]#tar-zxvfzookeeper-3.4.8.tar.gz-C/usr/local/[root@master1opt]#cd/usr/lib[root@master1lib]#mvjdk1.8.0_341jdk1.8.0[root@master1lib]#cd/usr/local/[root@master1local]#mvhadoop-2.7.7hadoop[root@master1local]#mvzookeeper-3.4.8/zookeeper6.編輯“~/.bashrc”文件，將相應(yīng)軟件的安裝路徑添加到環(huán)境變量中，操作內(nèi)容如下。exportJAVA_HOME=/usr/lib/jdk1.8.0exportHADOOP_HOME=/usr/local/hadoopexportZOOKEEPER_HOME=/usr/local/zookeeperPATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$ZOOKEEPER_HOME/bin7.配置免密登錄生成密鑰，配置免密登錄并測(cè)試，操作命令如下。[root@master1~]#ssh-keygen-trsa[root@master1~]#ssh-copy-idmaster1[root@master1~]#ssh-copy-idslave1[root@master1~]#ssh-copy-idslave2[root@master1~]#sshslave1Lastlogin:MonMar608:15:562023from[root@slave1~]#8.配置zookeeper切換到zookeeper下的conf目錄，通過復(fù)制“zoo_sample.cfg”文件創(chuàng)建“zoo.cfg”文件，操作命令如下。[root@master1conf]#cpzoo_sample.cfgzoo.cfg編輯“zoo.cfg”文件，操作命令及文件內(nèi)容如下。[root@master1conf]#vizoo.cfg

tickTime=2000initLimit=10syncLimit=5clientPort=2181dataDir=/usr/local/zookeeper/datadataLogDir=/usr/local/zookeeper/logsserver.1=master1:2888:3888server.2=slave1:2889:3889server.3=slave2:2890:3890其中配置項(xiàng)“dataDir”和“dataLogDir”中的目錄需要手動(dòng)創(chuàng)建，操作命令如下。[root@master1zookeeper]#mkdirdata[root@master1zookeeper]#mkdirlogs進(jìn)入新創(chuàng)建的data目錄中，創(chuàng)建myid文件，并編輯其內(nèi)容為1，操作命令如下。[root@master1zookeeper]#cddata/[root@master1data]#touchmyid9.配置Hadoop配置完Zookeeper后，切換到“/usr/local/hadoop/etc/hadoop”目錄中在此查看目錄，操作命令如下。[root@master1hadoop]#lscapacity-scheduler.xmlhttpfs-env.shmapred-env.shconfiguration.xsl

httpfs-pertiesmapred-queues.xml.templatecontainer-executor.cfg

httpfs-signature.secretmapred-site.xml.templatecore-site.xmlhttpfs-site.xmlslaveshadoop-env.cmdkms-acls.xmlssl-client.xml.examplehadoop-env.shkms-env.shssl-server.xml.examplepertiespertiesyarn-env.cmdpertieskms-site.xmlyarn-env.shhadoop-policy.xmlpertiesyarn-site.xmlhdfs-site.xmlmapred-env.cmd編輯hadoop-env.sh，修改JAVA_HOME配置項(xiàng)，配置內(nèi)容如下。#Thejavaimplementationtouse.exportJAVA_HOME=/usr/lib/jdk1.8.0在“/usr/local/hadoop/etc/hadoop”目錄中，通過拷貝“mapred-site.xml.template”文件生成“mapred-site.xml”文件，操作命令如下。[root@master1hadoop]#cpmapred-site.xml.templatemapred-site.xml分別編輯core-site.xml、hdfs-site.xml、mapred-site.xml，yarn-site.xml四文件個(gè)文件。<configuration><!--使用fs.defa還是使用fs.defaultFS，首先判斷是否開啟了namenode的高可用，如果開啟了就用fs.defaultFS，在單一namenode的情況下，就用fs.defa

<!--指定默認(rèn)的HDFS路徑--><property><name>fs.defaultFS</name><value>hdfs://mycluster</value></property><!--指定NameNode、DataNode、secondaryNamenode等存放數(shù)據(jù)的公共目錄--><property><name>hadoop.tmp.dir</name><value>/usr/local/hadoop/tmp</value></property><!--指定zookeeper地址--><property><name>ha.zookeeper.quorum</name><value>master1:2181,slave1:2181,slave2:2181</value></property><!--設(shè)置hadoop鏈接zookeeper的超時(shí)時(shí)間--><property><name>ha.zookeeper.session-timeout.ms</name><value>30000</value></property><!--設(shè)置垃圾回收時(shí)間，以分鐘為單位，1440分鐘=60*24，剛好是一天--><property><name>erval</name><value>1440</value></property></configuration>編輯“core-site.xml”文件，內(nèi)容如下。

（文本過長，放在ppt頁面下方）編輯“hdfs-site.xml”文件，文件內(nèi)容如下，（文本過長，放在ppt頁面下方，第一頁）。

<configuration><!--指定journalnode集群之間通信的超時(shí)時(shí)間--><property><name>dfs.qjournal.start-segment.timeout.ms</name><value>60000</value></property><!--指定namenode的組名稱，自己可以定義--><property><name>dfs.nameservices</name><value>mycluster</value></property><!--指定namenode組mycluster的成員--><property><name>dfs.ha.namenodes.mycluster</name><value>master1,slave1</value></property><!--配置master1的rpc地址及端口號(hào)--><property><name>dfs.namenode.rpc-address.mycluster.master1</name><value>master1:9000</value></property><!--配置slave1的rpc地址及端口號(hào)--><property><name>dfs.namenode.rpc-address.mycluster.slave1</name><value>slave1:9000</value></property><!--配置master1的地址及端口號(hào)--><property><name>dfs.namenode.http-address.mycluster.master1</name><value>master1:50070</value></property><!--配置slave1的地址及端口號(hào)--><property><name>dfs.namenode.http-address.mycluster.slave1</name><value>slave1:50070</value></property><!--指定NameNode的edits元數(shù)據(jù)在JournalNode上的存放位置--><property><name>dfs.namenode.shared.edits.dir</name><value>qjournal://master1:8485;slave1:8485;slave2:8485/mycluster</value></property><!--指定該集群出故障時(shí)，哪個(gè)實(shí)現(xiàn)類負(fù)責(zé)執(zhí)行故障切換--><property> <name>dfs.client.failvider.mycluster</name><value> org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider </value></property><!--配置隔離機(jī)制方法，多個(gè)機(jī)制用換行分割，即每個(gè)機(jī)制暫用一行--><property> <name>dfs.ha.fencing.methods</name><value>

sshfenceshell(/bin/true)</value></property><!--如果為"true"，則在HDFS中啟用權(quán)限檢查;如果為"false"，則關(guān)閉權(quán)限檢查;默認(rèn)值為"true"。--><property><name>dfs.permissions.enabled</name><value>false</value></property>

<property><!--指定是否允許支持文件追加，默認(rèn)為”false”--><name>dfs.support.append</name><value>true</value></property><!--使用sshfence隔離機(jī)制時(shí)需要ssh免登錄--><property><name>dfs.ha.fencing.ssh.private-key-files</name><value>/root/.ssh/id_rsa</value></property><!--數(shù)據(jù)備份的個(gè)數(shù)--><property><name>dfs.replication</name><value>2</value></property><!--指定hdfs元數(shù)據(jù)存儲(chǔ)的路徑--><property><name>dfs.namenode.name.dir</name><value>/usr/local/hadoop/dfs/name</value></property><!--指定hdfs數(shù)據(jù)存儲(chǔ)的路徑--><property><name>dfs.datanode.data.dir</name><value>/usr/local/hadoop/dfs/data</value></property><!--指定journalnode日志文件存儲(chǔ)的路徑--><property><name>dfs.journalnode.edits.dir</name><value>/usr/local/hadoop/tmp/journal</value></property><!--開啟自動(dòng)故障轉(zhuǎn)移--><property><name>dfs.ha.automatic-failover.enabled</name><value>true</value></property><!--開啟WebHDFS功能（基于REST的接口服務(wù)）--><property><name>dfs.webhdfs.enabled</name><value>true</value></property><!--配置sshfence隔離機(jī)制超時(shí)時(shí)間--><property><name>dfs.ha.fencing.ssh.connect-timeout</name><value>30000</value></property><!--設(shè)定從CLI手動(dòng)運(yùn)行的FC功能等待健康檢查、服務(wù)狀態(tài)超時(shí)時(shí)間--><property><name>ha.failover-controller.cli-check.rpc-timeout.ms</name><value>60000</value></property></configuration>編輯“hdfs-site.xml”文件，文件內(nèi)容如下，（文本過長，放在ppt頁面下方，第二頁）。

編輯“mapred-site.xml”文件，文件內(nèi)容如下。<configuration><!--設(shè)定指定mapreduce執(zhí)行框架為YARN方式--><property><name>map</name><value>yarn</value></property><!--設(shè)定jobhistory的IPC地址--><property><name>mapreduce.jobhistory.address</name><value>master1:10020</value></property><!--設(shè)定jobhistory的web地址--><property><name>mapreduce.jobhistory.webapp.address</name><value>master1:19888</value></property></configuration>編輯“yarn-site.xml”文件，文件內(nèi)容如下。（文本過長，放在ppt頁面下方）<configuration><!--開啟resourcemanger高可用--><property><name>yarn.resourcemanager.ha.enabled</name><value>true</value></property><!--指定resourcemanger的clusterid--><property><name>yarn.resourcemanager.cluster-id</name><value>yrc</value></property><!--指定resourcemanger的名字--><property><name>yarn.resourcemanager.ha.rm-ids</name><value>rm1,rm2</value></property><!--分別指定resourcemanger的地址--><property><name>yarn.resourcemanager.hostname.rm1</name><value>master1</value></property><!--分別指定resourcemanger的地址--><property><name>yarn.resourcemanager.hostname.rm2</name><value>slave1</value></property><!--指定resourcemanger使用ZK集群地址--><property><name>yarn.resourcemanager.zk-address</name><value>master1:2181,slave1:2181,slave2:2181</value></property><!--配置yarn運(yùn)行shuffle的方式，選擇mapreduce的默認(rèn)shuffle算法--><property><name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value></property><!--是否啟用日志聚集功能--><property><name>yarn.log-aggregation-enable</name><value>true</value></property><!--配置聚集的日志在HDFS上最多保存時(shí)間--><property><name>yarn.log-aggregation.retain-seconds</name><value>86400</value></property><!--開啟Recovery后，ResourceManger會(huì)將應(yīng)用的狀態(tài)等信息保存到y(tǒng)arn.resourcemanager.store.class配置的存儲(chǔ)介質(zhì)中，重啟后會(huì)load這些信息，并且NodeManger會(huì)將還在運(yùn)行的container信息同步到ResourceManager，整個(gè)過程不影響作業(yè)的正常運(yùn)行。--><property><name>yarn.resourcemanager.recovery.enabled</name><value>true</value></property><!--指定yarn.resourcemanager.store.class的存儲(chǔ)介質(zhì)(HA集群只支持ZKRMStateStore)--><property><name>yarn.resourcemanager.store.class</name><value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value></property></configuration>修改slaves文件(3.X為workers)，編輯內(nèi)容如下。master1slave1slave2在core-site.xml、hdfs-site.xml、mapred-site.xml，yarn-site.xml四個(gè)配置文件中，所涉及的配置項(xiàng)中的目錄需要手動(dòng)創(chuàng)建，操作命令如下。[root@master1hadoop]#mkdirlogs[root@master1hadoop]#mkdirdfs[root@master1hadoop]#mkdirtmp[root@master1hadoop]#cddfs[root@master1dfs]#mkdirdata[root@master1dfs]#mkdirname[root@master1dfs]#cd..[root@master1hadoop]#cdtmp/[root@master1tmp]#mkdirjournal在master1節(jié)點(diǎn)中分別拷貝“.bashrc”、“jdk”、“hadoop”和“zookeeper”文件到slave1和slave2節(jié)點(diǎn)中，操作命令如下。[root@master1~]#scp-r~/.bashrcslave1:~/.bashrc[root@master1~]#scp-r~/.bashrcslave2:~/.bashrc[root@master1~]#scp-r/usr/lib/jdk1.8.0slave1:/usr/lib/[root@master1~]#scp-r/usr/lib/jdk1.8.0slave2:/usr/lib/[root@master1~]#scp-r/usr/local/zookeeperslave1:/usr/local/[root@master1~]#scp-r/usr/local/zookeeperslave2:/usr/local/[root@master1~]#scp-r/usr/local/hadoopslave1:/usr/local/[root@master1~]#scp-r/usr/local/hadoopslave2:/usr/local/分別在slave1和slave2節(jié)點(diǎn)上執(zhí)行“source~/.bashrc”命令使環(huán)境變量立即生效。10.啟動(dòng)Hadoop集群（1）啟動(dòng)Zookeeper集群并查看狀態(tài)。在master1節(jié)點(diǎn)啟動(dòng)Zookeeper并查看狀態(tài)，操作命令及結(jié)果如下。[root@master1bin]#./zkServer.shstartZooKeeperJMXenabledbydefaultUsingconfig:/usr/local/zookeeper/bin/../conf/zoo.cfgStartingzookeeper...STARTED[root@master1bin]#./zkServer.shstatusZooKeeperJMXenabledbydefaultUsingconfig:/usr/local/zookeeper/bin/../conf/zoo.cfgMode:follower

在slave1節(jié)點(diǎn)啟動(dòng)Zookeeper并查看狀態(tài)，操作命令及結(jié)果如下。[root@slave1zookeeper]#cdbin[root@slave1bin]#./zkServer.shstartZooKeeperJMXenabledbydefaultUsingconfig:/usr/local/zookeeper/bin/../conf/zoo.cfgStartingzookeeper...STARTED[root@slave1bin]#./zkServer.shstatusZooKeeperJMXenabledbydefaultUsingconfig:/usr/local/zookeeper/bin/../conf/zoo.cfgMode:follower

在slave2節(jié)點(diǎn)啟動(dòng)Zookeeper并查看狀態(tài)，操作命令及結(jié)果如下。[root@slave2zookeeper]#cdbin[root@slave2bin]#./zkServer.shstartZooKeeperJMXenabledbydefaultUsingconfig:/usr/local/zookeeper/bin/../conf/zoo.cfgStartingzookeeper...STARTED[root@slave2bin]#./zkServer.shstatusZooKeeperJMXenabledbydefaultUsingconfig:/usr/local/zookeeper/bin/../conf/zoo.cfgMode:leader（2）初始化HA在Zookeeper中的狀態(tài)，操作命令及部分結(jié)果如下。[root@master1bin]#./hdfs

zkfc-formatZK23/03/0614:42:42INFOha.ActiveStandbyElector:Successfullycreated/hadoop-ha/myclusterinZK.23/03/0614:42:42INFOha.ActiveStandbyElector:Sessionconnected.23/03/0614:42:42INFOzookeeper.ZooKeeper:Session:0x386b7555fb80000closed23/03/0614:42:42INFOzookeeper.ClientCnxn:EventThreadshutdown（3）在master1、slave1和slave2中啟動(dòng)journalnode，操作命令如下。[root@master1sbin]#./hadoop-daemon.shstartjournalnode[root@slave1sbin]#./hadoop-daemon.shstartjournalnode[root@slave2sbin]#./hadoop-daemon.shstartjournalnode在master1中初始化namenode節(jié)點(diǎn)，如果看到“status0”字樣即代表初始化成功，操作命令及部分結(jié)果如下。[root@master1bin]#hdfs

namenode-format23/03/0614:47:20INFOutil.ExitUtil:Exitingwithstatus023/03/0614:47:20INFOnamenode.NameNode:SHUTDOWN_MSG:/************************************************************SHUTDOWN_MSG:ShuttingdownNameNodeatmaster1/10************************************************************/（4）在master1節(jié)點(diǎn)執(zhí)行‘start-all.sh’命令，并分別查看master1、slave1和slave2節(jié)點(diǎn)進(jìn)程，操作命令及結(jié)果如下。[root@master1hadoop]#start-all.sh[root@master1hadoop]#jps4896ResourceManager225QuorumPeerMain5330Jps3879JournalNode4072DFSZKFailoverController5016NodeManager3673DataNode3565NameNode

[root@slave1hadoop]#jps2322Jps1783DFSZKFailoverController218QuorumPeerMain1596DataNode1692JournalNode2190NodeManager

[root@slave2hadoop]#jps228QuorumPeerMain1173DataNode1269JournalNode1592NodeManager1724Jps（5）格式化主、從節(jié)點(diǎn)將master1節(jié)點(diǎn)元數(shù)據(jù)復(fù)制到slave1和slave2節(jié)點(diǎn)中，由于之前namenode，datanode的數(shù)據(jù)存放在hadoop/dfs目錄下，journalnode的數(shù)據(jù)存放在hadoop/tmp目錄下，所以直接復(fù)制dfs和tmp目錄至slave1和slave2節(jié)點(diǎn)中，操作命令如下。[root@master1hadoop]#scp-r/usr/local/hadoop/tmp/*slave1:/usr/local/hadoop/tmp/[root@master1hadoop]#scp-r/usr/local/hadoop/dfs/*slave1:/usr/local/hadoop/dfs/[root@master1hadoop]#scp-r/usr/local/hadoop/tmp/*slave2:/usr/local/hadoop/tmp/[root@master1hadoop]#scp-r/usr/local/hadoop/dfs/*slave2:/usr/local/hadoop/dfs/（6）在slave1節(jié)點(diǎn)啟動(dòng)resoucemanager和namenode，操作命令及結(jié)果如下。[root@slave1sbin]#./yarn-daemon.shstartresourcemanager[root@slave1sbin]#./hadoop-daemon.shstartnamenode[root@slave1