系統(tǒng)綜合應(yīng)用開發(fā)大作業(yè)

1、2016-2017學(xué)年第一學(xué)期系統(tǒng)綜合應(yīng)用開發(fā)大作業(yè)課題名稱：Hadoop集群搭建及MapReduce應(yīng)用班 級(jí)：13級(jí)軟工數(shù)據(jù)庫(kù)1班學(xué)號(hào):123456姓名： 123456成績(jī)：2016年10月、Hadoop集群的搭建與配置1、節(jié)點(diǎn)準(zhǔn)備此部分主要描述節(jié)點(diǎn)的規(guī)劃，包括IP的分配，NameNode和DataNode的配置。1、設(shè)置IP修改 /etc/sysconfig/network-scripts/ifcfg-eth0 文件：DEVICE=eth0 -網(wǎng)卡名稱BOOTPROTO=static -獲取靜態(tài) IPHWADDR=00:E0:4C:F8:3B:CE -機(jī)器 MAC 地址 IPADDR=4

2、0 -IPNETMASK= - 子 網(wǎng)掩碼 NETWORK=54 - 默認(rèn)網(wǎng) 關(guān) ONBOOT=yes2、設(shè)置 hostname在所有機(jī)器的/etc/hosts文件下添加：40 namenode46 datanode139 datanode249 datanode338 datanode451 datanode5注：為了使機(jī)器辨別到 hostname,必須重啟機(jī)器。2、Java環(huán)境安裝.進(jìn)入安裝目錄cd /home/magic/gshare/linux下構(gòu)造java開發(fā)環(huán)境 我的文件夾目錄為:/home/magic/gshare/limjx 下構(gòu)造java開發(fā)環(huán)境jdk-6u 134inux-

3、i 586. binrootmagtc - VirtualBox: /hone/riaglc# cd /hoEe/Eaqtc/gshdre八inux下第造av己開發(fā)環(huán)境 rootmagic - vtrtualBox: /ho(ne/nagtc/gshare/ltniixT J日va開發(fā)壞埃。.賦予權(quán)限sudo chmod u+x jdk-6u13-linux-i586.bin。(工如白研七 Virtual。*:/hoMe/naglc/gsh3E/llnux下構(gòu)造 Jav白開發(fā)環(huán)境 sudo chnod u+x 1dk6ui3-linuK-tSfi6+bln3、執(zhí)行安裝 sudo .Zjdk-6

4、u13-linux-i586.bin ,進(jìn)行編譯。oot的agtt-Vlrtual日8：/hg再e/Mdgtt/g寫Mr/Uu羯下構(gòu)造 java開發(fā)訃0V sudo , / jdk -6ul 3 “Iinux-t 586.bin Sun Microsystems, Inc. Binary Code License Agreementfor the JAVA 5E DEVELOPMENT KIT (JDK), VERSION! 6SUN MICROSYSTEMS, INJ (SUN) IS WILLING TO LICENSE THE SOFTWARE IDENTIFIED BELOW TO Y

5、OU OHLV UPON THE CONDITION THAT YOU 4c匚毛PT ALL OF THE TERMS CONTAINED IN THIS BINARY CODE LICENSE AGREEMENT 息ND SUPPLEMENT* LICENSE TERMS (COLLECTIVELY AGREEMENT ). PLEASE READ THE AGREEMENT CAREFULLY. 0V gHNLQADIWG OR INSTALLING THIS SOFTWARE, YOU ACCEPT THE TERMS OF THE AGREEMENT. INDICATE ACCEPTA

6、NCE BY SELECTING THE ACCEPT BUTTON AT THE BOTTOM OF THE AGREEMENT. IF YOU ARE NOT MILLIHG TO BE BOUND BY ALL THE TERMS. SELECT THE DECLINE BUTTON AT THE BOTTOM OF THE AGREEMENT AND THE DOWNLOAD OR INSTALL PROCESS HILL NOT CONTINUE,1. DEFINITIONS, software means the identified above in接著一直按回車鍵，直到出現(xiàn)選項(xiàng)

7、:Please enter T,yes or no”.Do you agree to the above license terns? yes or no輸入yes,按回車鍵。就可以完成安裝了。3、 SSH酉己置hadoop控制腳本依賴ssh來(lái)執(zhí)行針對(duì)整個(gè)集群的操作，因此為了支持無(wú)縫工作，此部分 主要描述SSH的配置。1、配置SSH(1)在namenode下執(zhí)行ssh-keygen -t rsa (2)在 namenode 下執(zhí)行ssh-copy-id -i /.ssh/id_rsa.pub rootnamenode ssh-copy-id -i /.ssh/id_rsa.pub rootda

8、tan ode1 ssh-copy-id -i /.ssh/id_rsa.pub rootdatanode2 ssh-copy-id -i /.ssh/id_rsa.pub root datanode3 ssh-copy-id -i /.ssh/id_rsa.pub rootdatanode4 ssh-copy-id -i /.ssh/id_rsa.pub r ootdatanode5遇到輸入提示時(shí)，如果不是詢問(wèn)密碼，可直接按回車進(jìn)入下一步。執(zhí)行完成后，實(shí)際上 在 namenode 節(jié) 點(diǎn) 的 /.ssh 目 錄 下 生 成 了 三 個(gè) 文 件：authorized_keys、id_rsa、i

9、d_rsa.pub,在 datanode節(jié)點(diǎn)下分另1J生成了 authorized_keys 文件，作為密鑰實(shí)現(xiàn)無(wú)密碼通信 (2)判斷是否安裝成功在namenode節(jié)點(diǎn)執(zhí)行ssh datanode1,如果沒有詢問(wèn)密碼就進(jìn)入 datanode1的命令行界面, 證明安裝成功，其他節(jié)點(diǎn)類似。rootljnamenode # ssh datanodelLasc login: Thu Jul 11 09:21:09 2013 from namenode | rootSdaranodelT * 2、安裝JDK進(jìn)入jdk-6u34-linux-i586-rpm.bin 文件所在目錄，執(zhí)行以下命令進(jìn)行安 裝：

10、.Zjdk-6u34-linux-i586-rpm.bin(2)安裝完成后在Zetc/profile文件下加入JDK環(huán)境變量：#config javaexport JAVA_HOME=/usr/java/jdk1.6.0_34export CLASSPATH=.:$JA VA_HOME/lib/tools.jar:$JA VA_HOME/lib/dt.jar export PATH=$J AVA_HOME/bin:$PA TH(3)使設(shè)置生效source /etc/profile4、Hadoop 配置1）配置 hadoop-env.sh 該hadoop-env.sh文件位于 /usr/hado

11、op/etc/hadoop目 錄下。 vim /usr/hadoop/etc/hadoop/hadoop-env.sh 在文件的末尾添力口下面內(nèi)容。# set java environmentexport JAVA_HOME=/usr/java/jdk1.8.0_05 2）配置 yarn-env.sh 修改 JAVA_HOME 值 export JAVA_HOME=/usr/java/jdk1.8.0_05 / 3 ）配置 slaves 文件 添加以下內(nèi)容：（為數(shù)據(jù)節(jié)點(diǎn)服務(wù)器相應(yīng)的IP地址） 并將原有的localhost行去掉。4）配置 core-site.xml文件 修改Hadoop核心配置

12、文件 core-site.xml,這里配置的是 HDFS的地址 和端 口號(hào)。文件修改后的內(nèi)容如下： property hadoop.tmp.dir/usr/hadoop/tmpA base for other temporary directories.fs.defaultFShdfs:/:9000 備注：如沒有配置hadoop.tmp.dir參數(shù)，此時(shí)系統(tǒng)默認(rèn)的臨時(shí)目錄為：/tmp/hadoo-hadoop。而這個(gè)目錄在每次重啟后都會(huì)被干掉，必須重新執(zhí)行 format 才行，否則會(huì)出錯(cuò)。5）配置hdfs-site.xml文件 文件修改后的內(nèi)容如下： configuration） node.s

13、econdary.http-address:9001.dirfile:/usr/hadoop/dfs/namedfs.datanode.data.dirfile:/usr/hadoop/dfs/datadfs.replication1dfs.webhdfs.enabledtrue 6）配置mapred-site.xml 文件 將 mapred-queues.xml.template 復(fù)制成 mapred-site.xml 文件，修改 Hadoop中 MapReduce的配置文件，配置的是JobTracker的地址和端口。修改后的mapred-site.xml 文件內(nèi)容如下： yarn 7）配置

14、 yarn-site.xml文件修改后該文件的內(nèi)容如下：yarn.nodemanager.aux-services mapreduce_shuffle yarn.nodemanager.aux-services.mapreduce.shuffle.class ort.apache.hadoop.mapred.ShuffleHandleryarn.resourcemanager.address:8032yarn.resourcemanager.scheduler.address :8030yarn.resourcemanager.resource-tracker.address :8031yar

15、n.resourcemanager.admin.address :8033yarn.resourcemanager.webapp.address :8088 5、Hadoop 測(cè)試此部分主要對(duì)Hadoop進(jìn)行測(cè)試。啟動(dòng)yarn在主節(jié)點(diǎn)上執(zhí)行cd /opt/moudles/hadoop-2.7.1/ ./sbin/start-yarn.shshell!yarn( HYPERLINK /20160613155223649 /20160613155223649)#驗(yàn)證安裝成功#瀏覽器查看 通過(guò)瀏覽器訪問(wèn) HYPERLINK http:/hadoop01:50070 http:/hadoop01:50

16、070 ! 瀏覽器( HYPERLINK /20160613155409636 /20160613155409636) 通過(guò)瀏覽器訪問(wèn) HYPERLINK http:/hadoop01:8088 http:/hadoop01:8088 ! 瀏覽器( HYPERLINK /20160613155503215 /20160613155503215)#程序驗(yàn)證 執(zhí)行如下代碼運(yùn)行帶有12個(gè)map和100個(gè)樣本的pi實(shí)例shell cd /opt/moudles/hadoop-2.7.1/share/hadoop/mapreduce yarnjar ./hadoop-mapreduce-examples

17、-2.7.1.jar pi 12 100執(zhí)行結(jié)果如下所示：Number of Maps = 12Samples per Map = 100Wrote input for Map #0Wrote input for Map #1Wrote input for Map #2Wrote input for Map #3Wrote input for Map #4Wrote input for Map #5Wrote input for Map #6Wrote input for Map #7Wrote input for Map #8Wrote input for Map #9Wrote input

18、 for Map #10Wrote input for Map #11Starting Job16/06/11 17:07:12 INFO client.RMProxy: Connecting toResourceManager at hadoop01/1:803216/06/11 17:07:12 INFO input.FileInputFormat: Total input paths to process : 1216/06/11 17:07:12 INFO mapreduce.JobSubmitter: number of splits:1216/06/11 17:07:12 INFO

19、 mapreduce.JobSubmitter: Submitting tokens for job: job_1465618407612_000616/06/11 17:07:13 INFO impl.YarnClientImpl: Submitted application application_1465618407612_000616/06/11 17:07:13 INFO mapreduce.Job: The url to track the job: HYPERLINK http:/hadoop01:8088/proxy/application_1465618407612_0006

20、/ http:/hadoop01:8088/proxy/application_1465618407612_0006/ 16/06/11 17:07:13 INFO mapreduce.Job: Running job: job_1465618407612_000616/06/11 17:07:17 INFO mapreduce.Job: Job job_1465618407612_0006 running in uber mode : false 16/06/11 17:07:17 INFO mapreduce.Job: map 0% reduce 0% 16/06/11 17:07:29

21、INFO mapreduce.Job: map 8% reduce 0% 16/06/11 17:07:30 INFO mapreduce.Job: map 67% reduce 0% 16/06/11 17:07:36 INFO mapreduce.Job: map 75% reduce 0% 16/06/11 17:07:37 INFO mapreduce.Job: map 100% reduce 100% 16/06/11 17:07:37 INFO mapreduce.Job: Job job_1465618407612_0006 completed successfully16/06

22、/11 17:07:37 INFO mapreduce.Job: Counters: 49File System CountersFILE: Number of bytes read=270FILE: Number of bytes written=1505992FILE: Number of read operations=0FILE: Number of large read operations=0FILE: Number of write operations=0HDFS: Number of bytes read=3170HDFS: Number of bytes written=2

23、15HDFS: Number of read operations=51HDFS: Number of large read operations=0HDFS: Number of write operations=3Job CountersLaunched map tasks=12Launched reduce tasks=1Data-local map tasks=12Total time spent by all maps in occupied slots (ms)=157380Total time spent by all reduces in occupied slots (ms)

24、=5029Total time spent by all map tasks (ms)=157380Total time spent by all reduce tasks (ms)=5029Total vcore-seconds taken by all map tasks=157380Total vcore-seconds taken by all reduce tasks=5029Total megabyte-seconds taken by all map tasks=161157120Total megabyte-seconds taken by all reduce tasks=5

25、149696 Map-Reduce FrameworkMap input records=12Map output records=24Map output bytes=216Map output materialized bytes=336Input split bytes=1754Combine input records=0Combine output records=0Reduce input groups=2Reduce shuffle bytes=336Reduce input records=24Reduce output records=0Spilled Records=48S

26、huffled Maps =12Failed Shuffles=0Merged Map outputs=12GC time elapsed (ms)=3029CPU time spent (ms)=35170Physical memory (bytes) snapshot=3409559552Virtual memory (bytes) snapshot=11427811328Total committed heap usage (bytes)=2604138496Shuffle ErrorsBAD_ID=0CONNECTION=0IO_ERROR=0WRONG_LENGTH=0WRONG_M

27、AP=0WRONG_REDUCE=0File Input Format CountersBytes Read=1416File Output Format CountersBytes Written=97Job Finished in 25.365 secondsEstimated value of Pi is 3.14666666666666666667關(guān)閉hadoop停止yarn在主節(jié)點(diǎn)上執(zhí)行如下命令cd /opt/moudles/hadoop-2.7.1/ ./sbin/stop-yarn.sh停止hdfs在主節(jié)點(diǎn)執(zhí)行如下命令cd /opt/moudles/hadoop-2.7.1/./

28、sbin/stop-dfs.shMapReduce 應(yīng)用1、應(yīng)用描述對(duì)輸入文件中數(shù)據(jù)進(jìn)行就算學(xué)生平均成績(jī)。輸入文件中的每行內(nèi)容均為一個(gè)學(xué)生的姓名和他相應(yīng)的成績(jī)，如果有多門學(xué)科，則每門學(xué)科為一個(gè)文件。 要求在輸出中每行有兩個(gè)間隔的數(shù) 據(jù)，其中，第一個(gè)代表學(xué)生的姓名，第二個(gè)代表其平均成績(jī)。2、數(shù)據(jù)準(zhǔn)備7 7 6 67 6 8 7 六六六六 -IT - -IT - -z - -IT 6 6 4 26 9 8-8五五五五王王王干9 9 2 09 8 8 9 四四四四 8 8 0 28 7 8 8樣本輸入：nath：china：english：樣本偏出：3、設(shè)計(jì)思路計(jì)算學(xué)生平均成績(jī)是一個(gè)仿 WordCo

29、unt例子，用來(lái)重溫一下開發(fā) MapReduce程序的流 程。程序包括兩部分的內(nèi)容：Map部分和Reduce部分，分別實(shí)現(xiàn)了 map和reduce的功能。Map處理的是一個(gè)純文本文件，文件中存放的數(shù)據(jù)時(shí)每一行表示一個(gè)學(xué)生的姓名和他相應(yīng)一科成績(jī)。Mapper處理的數(shù)據(jù)是由InputFormat分解過(guò)的數(shù)據(jù)集，其中 InputFormat的作用 是將數(shù)據(jù)集切割成小數(shù)據(jù)集InputSplit ,每一個(gè)InputSlit將由一個(gè)Mapper負(fù)責(zé)處理。此外，InputFormat 中還提供了一個(gè) RecordReader 的實(shí)現(xiàn)，并將一個(gè)InputSplit 解析成 對(duì)提供給了 map函數(shù)。InputF

30、ormat的默認(rèn)值是 TextInputFormat ,它針對(duì)文本文件， 按行將 文本切割成 InputSlit ,并用 LineRecordReader 將 InputSplit 解析成 對(duì)，key 是行 在文本中的位置，value是文件中的 一行。 Map的結(jié)果會(huì)通過(guò) partion分發(fā)到Reducer, Reducer做完Reduce操作后，將通過(guò)以格式OutputFormat輸出。 Mapper最終處理的結(jié)果對(duì) ,會(huì)送至ij Reducer中進(jìn)行合并，合并的時(shí)候，有相同key的鍵/值對(duì)則送到同一 個(gè) Reducer上。Reducer是所有用戶定制 Reducer類地基礎(chǔ)，它的輸入是ke

31、y和這個(gè)key對(duì)應(yīng)的所有value的一個(gè)迭代器，同時(shí)還有 Reducer的上下文。Reduce的結(jié)果由Reducer.Context 的write方法輸出到文件中。4、程序代碼/ 輸出姓名和成績(jī) context.write(name, new IntWritable(scoreInt); public/ 實(shí)現(xiàn) reduce context) throwsstatic class Reduce extends Reducer 函數(shù) public void reduce(Text key, Iterable values, ContextIOException, InterruptedExcept

32、ion int sum = 0; int count = 0; Iterator iterator = values.iterator(); while (iterator.hasNext() sum += iterator.next().get(); 計(jì)算總分 count+;/統(tǒng)計(jì)總的科目數(shù) int average = (int) sum / count;/計(jì)算平均成績(jī)context.write(key, new IntWritable(average); public static void main(String args) throws Exception Configuration conf = new Configuration(); / 這句話彳艮關(guān)鍵conf.set(mapred.job.tracker, :9001); String ioArgs = new String score_in, score_out ; String otherArgs = new GenericOptionsParser(conf, ioArgs).getRemainingArgs(); if (otherArgs.length != 2) System.err.println(Usage: Score Average ); Sy