Hadoop安裝部署文檔

1.系統(tǒng)版本版本機(jī)器masterLinuxhadooopl2.6.32-431.el6.x86_64#1SMPSunNov1022:19:54EST2013x86_64x86_64x86_64GNU/Linux機(jī)器slaveLinuxhadooop12.6.32-431.el6.x86_64#1SMPSunNov1022:19:54EST2013x86_64x86_64x86_64GNU/LinuxHadopphadoop-2.5.22.環(huán)境規(guī)劃IpUser/pwdHostnme45Hadoop/123123Hmaster45Hadoop/123123HslaveF面將在Hmaster所執(zhí)行的命令使用黑色框,Hslave執(zhí)行的命令使用灰色框.3.分別使用root用戶，設(shè)置兩臺機(jī)器的主機(jī)名在Hmasterecho“kernel.hostname=Hmaster”>>/etc/sysctl.conf保存設(shè)置，并檢查：/sbin/sysctl-phostnameHmastervi/etc/sysconfig/networkHOSTNAME=Hmaster然后，設(shè)置主機(jī)名和IP對應(yīng)關(guān)系：vi/etc/hosts45Hmaster46Hslave在Hslaveecho“kernel.hostname=Hslave”>>/etc/sysctl.conf保存設(shè)置，并檢查：/sbin/sysctl-phostnameHslave然后，設(shè)置主機(jī)名和IP對應(yīng)關(guān)系：vi/etc/hostsHmasterHslave4.分別創(chuàng)建用戶Hmaster、Hslave在Hmaster使用root用戶創(chuàng)建Hadoop組，Hadoop用戶groupaddHadoopuseradd-gHadoopHadooppasswdHadoop在Hslave使用root用戶創(chuàng)建Hadoop組，Hadoop用戶groupaddHadoopuseradd-gHadoopHadooppasswdHadoop.設(shè)置兩臺機(jī)器ssh無密碼登錄一般系統(tǒng)是默認(rèn)安裝了ssh命令的，如果沒有，自行安裝。在Hmaster，切換到Hadoop用戶第一步：產(chǎn)生密鑰$ssh-keygen-tdsa-P''-f~/.ssh/id_dsaGeneratingpublic/privatedsakeypair.Createddirectory'/home/Hadoopm/.ssh'.Youridentificationhasbeensavedin/home/Hadoopm/.ssh/id_dsa.Yourpublickeyhasbeensavedin/home/Hadoopm/.ssh/id_dsa.pub.Thekeyfingerprintis:6e:96:18:54:e0:01:0f:09:d8:ba:9b:87:40:c3:d8:6dHadoopm@HmasterThekey,srandomartimageis:+--[DSA1024]第二步：導(dǎo)入authorized_keys$cat~/.ssh/id_dsa.pub>>~/.ssh/authorized_keys$chmod600authorized_keys第三步：ssh無密碼連接測試$sshHmaster第一次登錄需要密碼，exit退出后，重新登錄不需要☆如果出現(xiàn)提示Theauthenticityofhost'hmaster(192.15S.124,145)'can'tbeestablished.RSAkeyfingerprintis49:3c:fe：0b:a6:23:7b:64:36:93:7b:0a:e5:69:de.yousu「eym」w己「tt口continueconnectinq |$ssh-oStrictHostKeyChecking=noHmaster同樣，在Hslave，切換到Hadoop用戶產(chǎn)生密鑰$ssh-keygen-tdsa-P''-f~/.ssh/id_dsaGeneratingpublic/privatedsakeypair.Youridentificationhasbeensavedin/home/Hadoops/.ssh/id_dsa.Yourpublickeyhasbeensavedin/home/Hadoops/.ssh/id_dsa.pub.Thekeyfingerprintis:a6:c2:c8:17:76:95:8a:9a:3d:c9:f1:61:67:2d:09:c5Hadoops@HslaveThekey,srandomartimageis:+--[DSA1024]| .. |$cat~/.ssh/id_dsa.pub>>~/.ssh/authorized_keys$chmod600authorized_keysssh無密碼連接測試$sshHslave第四步：設(shè)置Hmaster與Hslave機(jī)器相互無密訪問進(jìn)入Hmaster機(jī)器Hadoop用戶的.ssh目錄（Hadoop用戶）$scpauthorized_keysHadoop@Hslave:~/.ssh/authorized_keys_from_master輸入Hslave機(jī)器Hadoop的密碼（Hadoop用戶）Hadoops@hslave'spassword:authorized_keys 100%6050.6KB/s 00:00進(jìn)入Hslave機(jī)器Hadoop的.ssh目錄（Hadoop用戶）$catauthorized_keys_from_master>>authorized_keys$scpauthorized_keysHadoop@Hmaster:~/.ssh/authorized_keys_from_slave輸入Hmaster機(jī)器Hadoop的密碼（Hadoop用戶）Hadoop@hmaster'spassword:authorized_keys 100%12071.2KB/s 00:00進(jìn)入Hmaster機(jī)器Hadoop用戶的.ssh目錄（Hadoop用戶）$catauthorized_keys_from_slave>>authorized_keys第五步：互相測試無密登錄在Hmaster機(jī)器（Hadoop用戶）$sshHslaveLastlogin:TueNov123:55:422016fromhmaster成功。在Hslave機(jī)器（Hadoop用戶）$sshHmasterLastlogin:TueNov123:56:132016fromhslave成功。.安裝jdk上傳jdk到Hmaster及Hslave機(jī)器，并執(zhí)行以下命令（Hadoop用戶）$tar-zxvfjdk-7u75-linux-x64.tar.gz復(fù)制jdk解壓路徑，設(shè)置Hadoop用戶環(huán)境變量$vi~/.bash_profileexportJAVA_HOME=/home/Hadoop/jdk1.7.0_75exportCLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jarexportPATH=$JAVA_HOME/bin:$PATH.安裝hadoop-2.5.2.解壓在Hmaser機(jī)器，解壓hadoop-2.5.2.tar.gz,并創(chuàng)建以下文件夾（Hadoop用戶）$tar-zxvfhadoop-2.5.2.tar.gz$mkdir~/dfs/$mkdir~/dfs/name$mkdir?/dfs/data$mkdir~/tmp.修改配置文件進(jìn)入/home/Hadoop/hadoop-2.5.2/etc/hadoop目錄（Hadoop用戶）$cd/home/Hadoop/hadoop-2.5.2/etc/hadoop修改core-site.xml$vicore-site.xml<configuration>(property〉<name>hadoop.tmp.dir</name><value>/home/Hadoop/tmp</value><description>Abaseforothertemporarydirectories.</description></property><property><name>fs.defaultFS</name><value>hdfs://Hmaster:9000</value></property><property><name>io.file.buffer.size</name><value>4096</value></property></configuration>修改hdfs-site.xml$vihdfs-site.xml<configuration><property><name>services</name><value>hadoop-cluster1</value></property><property><name>node.secondary.http-address</name><value>Hmaster:50090</value></property><property><name>.dir</name><value>file:///home/Hadoop/dfs/name</value>〈/property〉(property〉<name>dfs.datanode.data.dir</name><value>file:///home/Hadoop/dfs/data</value></property><property><name>dfs.replication</name><value>1</value></property><property><name>dfs.webhdfs.enabled</name><value>true</value></property></configuration>修改mapred-site.xml$cpmapred-site.xml.templatemapred-site.xml$vimapred-site.xml<configuration><property><name></name><value>yarn</value></property><property><name>mapreduce.jobtracker.http.address</name><value>Hmaster:50030</value></property><property><name>mapreduce.jobhistory.address</name><value>Hmaster:10020</value></property><property><name>mapreduce.jobhistory.webapp.address</name><value>Hmaster:19888</value>〈/property〉</configuration>修改yarn-site.xml<configuration>(property〉<name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value></property><property><name>yarn.resourcemanager.address</name><value>Hmaster:8032</value></property><property><name>yarn.resourcemanager.scheduler.address</name><value>Hmaster:8030</value></property><property><name>yarn.resourcemanager.resource-tracker.address</name><value>Hmaster:8031</value></property><property><name>yarn.resourcemanager.admin.address</name><value>Hmaster:8033</value></property><property><name>yarn.resourcemanager.webapp.address</name><value>Hmaster:8088</value></property></configuration>8088為訪問hadoop端口修改slaves$vislavesHslave修改hadoop-env.sh$vihadoop-env.shexportJAVA_HOME=/home/Hadoop/jdk1.7.0_75修改yarn-env.sh$viyarn-env.shexportJAVA_HOME=/home/Hadoop/jdk1.7.0_753.將hadoop-2.5.2文件夾傳輸?shù)紿slave$scp?/hadoop-2.5.2Hadoop@Hslave:?/.格式化文件系統(tǒng)在機(jī)器Hmaster及Hslave$/home/Hadoop/hadoop-2.5.2/bin/hdfsnamenode-format注意：這里的格式化文件系統(tǒng)并不是硬盤格式化，只是針對主服務(wù)器hdfs-site.xml的.dir和dfs.datanode.data.dir目錄做相應(yīng)的清