在VM下hadoop集群搭建

-.z〔原創(chuàng)〕VM下搭建hadoop集群前期準(zhǔn)備1.安裝VMware_Workstation_wmb2.安裝三臺CentOS-6.3-i386-bin-DVD1安裝步驟：〔在安裝centos時就把pc的名字改好，免得后面改hostname〕1.在每臺pc上的/etc/hosts中參加：localhost74master71slave173slave22.在每臺pc安裝java：在/etc/profile中參加：e*portPATH=:$JAVA_HOME/bin:/sbin:/usr/bin:/usr/sbin:/bine*portCLASSPATH=.:$JAVA_HOME/lib3.配置SSH無密碼登錄：在每臺pc上面：ssh-keygen-trsacat~/.ssh/id_rsa.pub>>~/.ssh/authorized_keyschmod700.sshchmod644authorized_keys編輯sshd配置文件/etc/ssh/sshd_config，把*AuthorizedKeysFile.ssh/authorized_keys前面的注釋取消掉。在master上面：ssh-copy-id-i~/.ssh/id_rsa.pubrootslave1ssh-copy-id-i~/.ssh/id_rsa.pubrootslave2重啟servicesshdrestart在slave1上面：ssh-copy-id-i~/.ssh/id_rsa.pubrootmasterssh-copy-id-i~/.ssh/id_rsa.pubrootslave2重啟servicesshdrestart在slave2上面：ssh-copy-id-i~/.ssh/id_rsa.pubrootslave1ssh-copy-id-i~/.ssh/id_rsa.pubrootmaster重啟servicesshdrestart再sshslave1;sshslave2;sshmaster驗(yàn)證**遇到Agentadmittedfailuretosignusingthekey

解決方式使用ssh-add指令將私鑰加進(jìn)來〔根據(jù)個人的密匙命名不同更改id_rsa〕

*ssh-add~/.ssh/id_rsa**遇到ssh:connecttohostmasterport22:Noroutetohost是由于ip地址問題，檢查/etc/hosts4.配置hadoop下載hadoop后scphadoop-.tar.gzrootslave1:/usr/localtar*zvfhadoop-.tar.gzmvhadoop-/usr/local/hadoop在profile和./bashrc中參加：e*portHADOOP_HOME=/usr/local/hadoope*portPATH=$PATH:$HADOOP_HOME/bin配置hadoop-env.sh設(shè)置JAVA_HOME配置core-site.*ml<property><name></name><value>hdfs://master:9000</value><final>true</final></property><property><name>hadoop.tmp.dir</name><value>/home/hadoop/hadoop/data/hdfs/tmp</value></property>配置mapred-site.*ml<property><name>mapred.job.tracker</name><value>master:9001</value></property>配置hdfs-site.*ml<!--master配--><property><name>.dir</name><value>/home/hadoop/hadoop/data/hdfs/name</value><final>true</final></property><!--slave配--><property><name>dfs.data.dir</name><value>/home/hadoop/hadoop/data/hdfs/data</value><final>true</final></property><property><name>dfs.replication</name><value>1</value><final>true</final></property>配置masters修改為master或?qū)?yīng)的IP地址配置slavesslave1slave2〔可以salves機(jī)上通過scp拷masters機(jī)上的hadoop主目錄：sudoscp-rtest0:/usr/local/hadoop/usr/local〕關(guān)掉防火墻root用戶下serviceiptablesstop格式化namenode在master機(jī)子中：hadoopnamenode-format在master下啟動集群即可：start-all.sh〔因?yàn)橐呀?jīng)將hadoop添加到PATH中去了〕檢驗(yàn)master：輸入jps3711NameNode4085Jps3970JobTracker3874SecondaryNameNode檢驗(yàn)slave：輸入jps2892Jps2721DataNode2805TaskTracker或在master輸入：hadoopdfsadmin-reportSafemodeisONConfiguredCapacity:(14.42GB)PresentCapacity:(12.79GB)DFSRemaining:(12.53GB)DFSUsed:276422656(263.62MB)DFSUsed%:2.01%Underreplicatedblocks:0Blockswithcorruptreplicas:0Missingblocks:0-------------------------------------------------Datanodesavailable:3(3total,0dead)Name:43:50010DemissionStatus:NormalConfiguredCapacity:5160566784(4.81GB)DFSUsed:41160704(39.25MB)NonDFSUsed:582455296(555.47MB)DFSRemaining:4536950784(4.23GB)DFSUsed%:0.8%DFSRemaining%:87.92%Lastcontact:MonMay0616:12:02CST2013Name:40:50010DemissionStatus:NormalConfiguredCapacity:5160566784(4.81GB)DFSUsed:97075200(92.58MB)NonDFSUsed:582545408(555.56MB)DFSRemaining:4480946176(4.17GB)DFSUsed%:1.88%DFSRemaining%:86.83%Lastcontact:MonMay0616:12:01CST2013Name:41:50010DemissionStatus:NormalConfiguredCapacity:5160566784(4.81GB)DFSUsed:138186752(131.79