講稿hadoop220集群部署

0 .-1Hadoop集群節(jié)點規(guī)劃與配置-2-創(chuàng)建Hadoop3-ssh免登錄配置-3JDK安裝配置4-Hadoop集群安裝配置4-配置etc/hadoop/hadoop-env.sh5-配置etc/hadoop/core-site.xml6-配置etc/hadoop/hdfs-site.xml8-配置etc/hadoop/yarn-env.sh9-配置etc/hadoop/yarn-site.xml11-配置etc/hadoop/mapred-env.sh13-配置etc/hadoop/mapred-site.xml13-配置etc/hadoop/slaves15-同步分發(fā)安裝文件15-Hadoop集群啟動驗證15-檢查狀態(tài)-15格式化HDFS15-啟動HDFS16-啟動YARN集群16-16-Hadoop集群節(jié)點規(guī)劃與配Hadoop集群點的配置信息以及各個節(jié)點的角色整理如下表所（主機名命名規(guī)則：KaolaStatistics的前兩個字母縮寫：IP節(jié)點角色（進程 、上表點角色（進程）縮寫說明，如下表所示HDFSHDFSSecondaryHDFSCheckpointHBaseHBaseetc/hosts文件，增加如下主機映射需要以超級用戶權限，修改所有節(jié)點對應的主機名稱。以主節(jié)點ks01創(chuàng)建Hadoop用useraddpasswd在Linux系統(tǒng)上，為Hadoop集群創(chuàng)建一useraddpasswdssh免登錄配ssh-keygen-tssh-keygen-t 下面生成了公鑰文件id_rsa.pub和密鑰文件id_rsa。catcat~/.ssh/id_rsa.pub>> sshsshodod644scp~/.ssh/id_rsa.pubks02scp~/.ssh/id_rsa.pub這時，需要輸入從節(jié)點ks02的才能執(zhí)行拷貝。拷貝成功以后，登錄到從節(jié)點catcat~/.ssh/id_rsa.pub.master>>/ssh 就能登錄到ks02上，說明配置成功。sshodg-w如果是第一次生成RSA文件一般來說都是~/.ssh/authorized_keys的權限都是有問題的，wodg-wg-w644JDK安裝配ln-s/usr/local/jdk1.7.0_45/假設安 ln-s/usr/local/jdk1.7.0_45/exportJAVA_HOME=/usr/local/javaexportJAVA_HOME=/usr/local/javaexport..sourcesourceHadoop集群安裝配2.2.0.ta.gz lnln-s/usr/local/hadoop-2.2.0進入 劃一下Hadoop相關 ，包括name export設置Hadoop環(huán)境變量，修改~/.bashrc文件，增加HADOOP_HOME配置（假設Hadoop安裝 2export..sourcesource##SetHadoop-specificenvironmentvariables#Theonlyrequiredenvironmentvariableis Allothers#optional. Whenrunningadistributedconfigurationitisbestto#setJAVA_HOMEinthisfile,sothatitiscorrectlydefinedon#remote#Thejavaimplementationtoexport#Thejsvcimplementationtouse.Jsvcisrequiredtorunsecure#export #forfin$HADOOP_HOME/contrib/capacity-scheduler/*.jar;do if["$HADOOP_CLASSPATH"];then export #umamountofheaptouse,inMB.Defaultisexportexport#ExtraJavaruntime Emptyby#CommandspecificoptionsappendedtoHADOOP_OPTSwhen 幾個配置行即可, #Thefollowingappliestomultiplecommands(fs,dfs,fsck,distcpetc)exportHADOOP_ _OPTS="-Xmx512m$HADOOP_ #Onsecuredatanodes,usertorunthedatanodeasafterdrop exportHADOOP_SECURE_DN_USER=${HADOOP_SECURE_DN_USER}#Wherelogfilesarestored. $HADOOP_HOME/logsbydefault.exportHADOOP_LOG_DIR=/home/hadoop/storage/logs#Wherelogfilesarestoredinthesecuredataexport#Thedirectorywherepidfilesarestored./tmpby#NOTE:thisshouldbesettoadirectorythatcanonlybewrittento theuserthatwillrunthehadoopdaemons. Otherwisethereisthe potentialforasymlinkattack.exportexport#Astringrepresentingthisinstanceofhadoop.$USERbydefault.exportHADOOP_IDENT_STRING=$USER exportHADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"<description>Default'file:///'.Thenameofthedefaultfilesystem.AURIandauthoritydeterminetheFileSystemimplementation.Theuri'sschemedeterminestheconfigproperty(fs.SCHEME.impl)namingtheFileSystemimplementationclass.Theuri'sauthorityisusedtodeterminethehost,port,etc.fora<description>Default4096.Thesizeofbufferforuseinsequencefiles.Thesizeofthisbuffershouldprobablybeamultipleofhardwarepagesize(4096onInx86),anditdetermineshowmuchdataisbufferedduringreadandwriteoperations.<description>Thenumberofsecondsbetweentwoperiodic <description>Thesizeofthecurrenteditlog(inbytes)thattriggersaperiodiccheckpointevenifthefs.checkpoint.periodhasn't .Thedefaultblocksizefornewfiles,inbytes.Youcanusethefollowingsuffix(caseinsensitive):k(kilo),m(mega),g(giga),t(),p(peta),e(exa)tospecifythesize(suchas128k,512m,1g,etc.),Orprovidecompletesizeinbytes(suchas for128MB).<description>NameNodegetthenewestfsimagevia#Userfor#UserforYARN#resolvelinks-$0maybea#someJavaif["$JAVA_HOME"!=""];then#echo"runjavainif["$JAVA_HOME"=""];echo"Error:JAVA_HOMEisnotset."exit1#ForsettingYARNspecificHEAPsizespleaseusethis#Parameterandsetappropriay##checkenvvarswhichmightoverridedefaultargsif["$YARN_HEAPSIZE"!=""];then#ResourceManagerspecific#SpecifythemaxHeapsizefortheResourceManagerusinganumericalvalue#inthescaleofMB.Forexample,tospecifyanjvmoptionof-Xmx1000m,set#thevalueto1000.#ThisvaluewillbeoverriddenbyanXmxsettingspecifiedineither#and/or#Ifnotspecified,thedefaultvaluewillbepickedfromeitherYARN_HEAPMAX#orJAVA_HEAP_MAXwithYARN_HEAPMAXasthepreferredoptionofthetwo.#exportYARN_RESOURCEMANAGER_HEAPSIZE=1000#SpecifytheJVMoptionstobeusedwhenstartingtheResourceManager.#TheseoptionswillbeappendedtotheoptionsspecifiedasYARN_OPTS#andthereforemayoverrideanysimilarflagssetinYARN_OPTS#export#NodeManagerspecific#SpecifythemaxHeapsizefortheNodeManagerusinganumerical#inthescaleofMB.Forexample,tospecifyanjvmoptionof-Xmx1000m,set#thevalueto1000.#ThisvaluewillbeoverriddenbyanXmxsettingspecifiedineitherYARN_OPTS#and/orYARN_NODEMANAGER_OPTS.#Ifnotspecified,thedefaultvaluewillbepickedfromeitherYARN_HEAPMAX#orJAVA_HEAP_MAXwithYARN_HEAPMAXasthepreferredoptionofthetwo.#exportYARN_NODEMANAGER_HEAPSIZE=1000#SpecifytheJVMoptionstobeusedwhenstartingthe#TheseoptionswillbeappendedtotheoptionsspecifiedasYARN_OPTS#andthereforemayoverrideanysimilarflagssetinYARN_OPTS#export#sothatfilenamesw/spacesarehandledcorrectlyinloopsbelow#defaultlogdirectory&fileif["$YARN_LOG_DIR"=""];if["$YARN_LOGFILE"=""];#default fileforservice-levelauthorizationif["$YARN_ FILE"=""];then #restoreordinarybehaviourunsetIFSYARN_OPTS="$YARN_OPTS-Dyarn.log.dir=$YARN_LOG_DIR"YARN_OPTS="$YARN_OPTS-Dyarn.log.file=$YARN_LOGFILE"YARN_OPTS="$YARN_OPTS-Dyarn.home.dir=$ YARN_OPTS="$YARN_OPTS-Dyarn.id.str=$YARN_IDENT_STRING"YARN_OPTS="$YARN_OPTS-Dyarn.root.logger=${YARN_ROOT_LOGGER:-INFO,console}" <?xml<?xml<description>Default'0.0.0.0'.Thehostnameofthe<description>Default8192.Amountofphysicalmemory,inMB,thatcanbeallocatedfor<description>Default8.NumberofCPUcoresthatcanbeallocatedfor<description>Default1024.TheminimumallocationforeveryrequestattheRM,inMBs.Memoryrequestslowerthanthiswon'ttakeeffect,andthespecifiedvaluewillgetallocatedatminimum. <description>Default8192.The umallocationforeverycontainerrequestattheRM,inMBs.Memoryrequestshigherthanthiswon'ttakeeffect,andwillgetcappedtothisvalue.<description>Listofdirectoriestostorelocalizedfilesin.Anapplication'slocalizedfiledirectorywillbefoundin:Individualcontainers'workdirectories,calledcontainer_${contid},willbesubdirectoriesofthis.<description>Wheretostorecontainerlogs.Anapplication'slocalizedlogdirectorywillbefoundin${yarn.nodemanager.log-dirs}/application_${appid}.Individualcontainers'logdirectorieswillbebelowthis,indirectoriesnamedcontainer_{$contid}.Eachcontainerdirectorywillcontainthefilesstderr,stdin,andsysloggeneratedbythatcontainer.<description>Wheretoaggregatelogs<description>Thevalidservicenameshouldonlycontaina-zA-Z0-9_andcannotstartwithnumbers.</description>exportexportHADOOP_MAPRED_ROOT_LOGGER=INFO,RFA#exportHADOOP_JOB_HISTORYSERVER_OPTS=exportexportHADOOP_MAPRED_ROOT_LOGGER=INFO,RFA#exportHADOOP_JOB_HISTORYSERVER_OPTS=#Wherelogfilesarestored. $HADOOP_MAPRED_HOME/logsbydefault.exportHADOOP_MAPRED_LOG_DIR=/home/hadoop/storage/logs#exportHADOOP_JHS_LOGGER=INFO,RFA#HadoopJobSummarylogger.#exportHADOOP_MAPRED_PID_DIR=#Thepidfilesarestored./tmpbydefault.#exportHADOOP_MAPRED_IDENT_STRING=#Astringrepresentingthisinstanceof$USERby#exportHADOOP_MAPRED_NICENESS=#Theschedulingpriorityfordaemons.Defaultsto<?xml<?xml<description>Default'local'.TheruntimeframeworkforexecutingMapRedu