hadoop安裝前準(zhǔn)備工作

1、在安裝好的Ubuntu系統(tǒng)下添加具有sudo權(quán)限的用戶。rootnodeA:# sudo adduser zyxAdding user zyx .Adding new group zyx (1001) .Adding new user zyx (1001) with group zyx .Creating home directory /home/zyx .Copying files from /etc/skel .Enter new UNIX password: Retype new UNIX password: passwd: password updated successfullyCh

2、anging the user information for zyxEnter the new value, or press ENTER for the default Full Name : Cadduser: /usr/bin/chfn zyx exited from signal 2. Exiting.rootnodeA:#rootnodeA:# sudo usermod -G admin -a zyxrootnodeA:#建立SSH無(wú)密碼登陸（1）namenode上實(shí)現(xiàn)無(wú)密碼登陸本機(jī)zyxnodeA:$ ssh-keygen -t dsa -P -f /.ssh/id_dsaGen

3、erating public/private dsa key pair.Created directory /home/zyx/.ssh.Your identification has been saved in /home/zyx/.ssh/id_dsa.Your public key has been saved in /home/zyx/.ssh/id_dsa.pub.The key fingerprint is:65:2e:e0:df:2e:61:a5:19:6a:ab:0e:38:45:a9:6a:2b zyxnodeAThe keys randomart image is:+- D

4、SA 1024-+| | . | o . o | o . .+. |. . .S=. |.o o.=o |+. . o. |E. . . |. .o. . |+-+zyxnodeA:$ cat /.ssh/id_dsa.pub /.ssh/authorized_keyszyxnodeA:$（2）實(shí)現(xiàn)namenode無(wú)密碼登陸其他datanodehadoopnodeB:$ scp hadoopnodea:/home/hadoop/.ssh/id_dsa.pub /home/hadoophadoopnodeas password: id_dsa.pub 100% 602 0.6KB/s 00:00

5、 hadoopnodeB:$ cat id_dsa.pub .ssh/authorized_keyshadoopnodeB:$ sudo ufw disable復(fù)制JDK（jdk-6u20-linux-i586.bin）文件到linux利用F-Secure SSH File Transfer Trial 工具，直接拖拽jdk-6u20-linux-i586.bin的安裝和配置（1）安裝zyxnodeA:$ lsExamples jdkzyxnodeA:$ cd jdkzyxnodeA:/jdk$ lsjdk-6u20-linux-i586.binzyxnodeA:/jdk$ chmod a+x

6、 jdk*zyxnodeA:/jdk$ ./jdk*接下來(lái)顯示許可協(xié)議，然后選擇yes， 然后按Enter鍵，安裝結(jié)束。zyxnodeA:/jdk$ lsjdk1.6.0_20 jdk-6u20-linux-i586.bin（2）配置用rootnodeA:/home/zyx# vi .bashrc 打開bashrc， 然后在最后加入下面幾行：export JAVA_HOME=/home/zyx/jdk/jdk1.6.0_20export JRE_HOME=/home/zyx/jdk/jdk1.6.0_20/jreexport CLASS_PATH=$CLASSPATH:$JAVA_HOME/l

7、ib:$JRE_HOME/libexport PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH:$HOMR/binHadoop的安裝下載地址： HYPERLINK /apache-mirror/hadoop/core/hadoop-0.20.2/hadoop-0.20.2.tar.gz /apache-mirror/hadoop/core/hadoop-0.20.2/hadoop-0.20.2.tar.gz把hadoop-0.20.2.tar.gz 放到home/zyx/hadoop下，然后解壓該文件zyxnodeB:/hadoop$ tar -zvxf hado

8、op-0.20.2.tar.gz設(shè)置環(huán)境變量， 添加到home/zyx/.bashrczyxnodeA:$ vi .bashrcexport HADOOP_HOME=/home/zyx/hadoop/hadoop-0.20.2export PATH=$HADOOP_HOME/bin:$PATHHadoop的配置在conf/hadoop-env.sh中配置java環(huán)境export JAVA_HOME=/home/zyx/jdk/jdk/jdk1.6.0_20 配置conf/masters, slaves 文件， 只需要在nodename上配置。配置core-site.xml, hdfs-site

9、.xml, mapred-site.xml zyxnodeC:/hadoop-0.20.2/conf$ more core-site.xml # hdfs:/03:54310 hdfs:/03:9000 zyxnodeC:/hadoop-0.20.2/conf$ more hdfs-site.xml dfs.replication 1 zyxnodeC:/hadoop-0.20.2/conf$ more mapred-site.xml mapred.job.tracker# hdfs:/03:54320 hdfs:/03:9001 Hadoop的運(yùn)行（0） 格式化：zyxnodeC:/hado

10、op-0.20.2/bin$ hadoop namenode format（1）用jps查看進(jìn)程：zyxnodeC:/hadoop-0.20.2/bin$ jps31030 NameNode31488 TaskTracker31283 SecondaryNameNode31372 JobTracker31145 DataNode31599 Jps（2）查看集群狀態(tài)zyxnodeC:/hadoop-0.20.2/bin$ hadoop dfsadmin -reportConfigured Capacity: 304716488704 (283.79 GB)Present Capacity: 27

11、0065557519 (251.52 GB)DFS Remaining: 270065532928 (251.52 GB)DFS Used: 24591 (24.01 KB)DFS Used%: 0%Under replicated blocks: 0Blocks with corrupt replicas: 0Missing blocks: 0 -Datanodes available: 1 (1 total, 0 dead) Name: 03:50010Decommission Status : NormalConfigured Capacity: 304716488704 (283.79

12、 GB)DFS Used: 24591 (24.01 KB)Non DFS Used: 34650931185 (32.27 GB)DFS Remaining: 270065532928(251.52 GB)DFS Used%: 0%DFS Remaining%: 88.63%Last contact: Fri Apr 23 15:39:10 CST 2010（3）Stop 文件：zyxnodeC:/hadoop-0.20.2/bin$ stop-all.shstopping jobtrackerlocalhost: stopping tasktrackerstopping namenodel

13、ocalhost: stopping datanodelocalhost: stopping secondarynamenode運(yùn)行一個(gè)簡(jiǎn)單JAVA 程序先在本地磁盤建立兩個(gè)文件file01和file02cuijjstation1 $ echo Hello cuijj bye cuijj file01cuijjstation1 $ echo Hello Hadoop Goodbye Hadoop file022）在hdfs中建立一個(gè)input目錄cuijjstation1 $ hadoop dfs -mkdir input將file01和file02拷貝到hdfs的input目錄下zyxnod

14、eC:$ hadoop dfs -copyFromLocal /home/zyx/file0* input查看hdfs中有沒(méi)有input目錄zyxnodeC:$ hadoop dfs -lsFound 1 itemsdrwxr-xr-x - zyx supergroup 0 2010-04-23 16:40 /user/zyx/input查看input目錄下有沒(méi)有復(fù)制成功file01和file02zyxnodeC:$ hadoop dfs -ls inputFound 2 items-rw-r-r- 1 zyx supergroup 0 2010-04-23 16:40 /user/zyx/i

15、nput/file01-rw-r-r- 1 zyx supergroup 0 2010-04-23 16:40 /user/zyx/input/file02執(zhí)行wordcount（確保hdfs上沒(méi)有output目錄）zyxnodeC:/hadoop-0.20.2$ hadoop jar hadoop-0.20.2-examples.jar wordcount input output10/04/24 09:25:10 INFO input.FileInputFormat: Total input paths to process : 210/04/24 09:25:11 INFO mapred

16、.JobClient: Running job: job_201004240840_000110/04/24 09:25:12 INFO mapred.JobClient: map 0% reduce 0%10/04/24 09:25:22 INFO mapred.JobClient: map 100% reduce 0%10/04/24 09:25:34 INFO mapred.JobClient: map 100% reduce 100%10/04/24 09:25:36 INFO mapred.JobClient: Job complete: job_201004240840_00011

17、0/04/24 09:25:36 INFO mapred.JobClient: Counters: 1710/04/24 09:25:36 INFO mapred.JobClient: Job Counters 10/04/24 09:25:36 INFO mapred.JobClient: Launched reduce tasks=110/04/24 09:25:36 INFO mapred.JobClient: Launched map tasks=210/04/24 09:25:36 INFO mapred.JobClient: Data-local map tasks=210/04/

18、24 09:25:36 INFO mapred.JobClient: FileSystemCounters10/04/24 09:25:36 INFO mapred.JobClient: FILE_BYTES_READ=7910/04/24 09:25:36 INFO mapred.JobClient: HDFS_BYTES_READ=5010/04/24 09:25:36 INFO mapred.JobClient: FILE_BYTES_WRITTEN=22810/04/24 09:25:36 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=4110/0

19、4/24 09:25:36 INFO mapred.JobClient: Map-Reduce Framework10/04/24 09:25:36 INFO mapred.JobClient: Reduce input groups=510/04/24 09:25:36 INFO mapred.JobClient: Combine output records=610/04/24 09:25:36 INFO mapred.JobClient: Map input records=210/04/24 09:25:36 INFO mapred.JobClient: Reduce shuffle

20、bytes=8510/04/24 09:25:36 INFO mapred.JobClient: Reduce output records=510/04/24 09:25:36 INFO mapred.JobClient: Spilled Records=1210/04/24 09:25:36 INFO mapred.JobClient: Map output bytes=8210/04/24 09:25:36 INFO mapred.JobClient: Combine input records=810/04/24 09:25:36 INFO mapred.JobClient: Map

21、output records=810/04/24 09:25:36 INFO mapred.JobClient: Reduce input records=6查看運(yùn)行結(jié)果zyxnodeC:/hadoop-0.20.2$ hadoop fs -cat output/part-r-00000Goodbye 1Hadoop 2Hello 2bye 1cuijj 2MapReduce的安裝MapReduce程序的運(yùn)行 HYPERLINK /wiki/Running_Hadoop_On_Ubuntu_Linux_(Single-Node_Cluster)#Excursus:_Hadoop_Distrib

22、uted_File_System_.28HDFS.29 /wiki/Running_Hadoop_On_Ubuntu_Linux_(Single-Node_Cluster)#Excursus:_Hadoop_Distributed_File_System_.28HDFS.2911. 對(duì)于.java程序的hadoop編譯：rootnodeC:/home/zyx/hadoop-0.20.2# javac -classpath /home/zyx/hadoop-0.20.2/hadoop-0.20.2-core.jar:/home/zyx/hadoop-0.20.2/lib/commons-cli-

23、1.2.jar -d /home/zyx/wordcount_class /home/zyx/hadoop-0.20.2/src/examples/org/apache/hadoop/examples/WordCount.java12. 把 .class 文件生成 .jar 文件rootnodeC:/home/zyx/wordcount_class/org/apache/hadoop/examples# jar -cvf /home/zyx/wordcount.jar /home/zyx/wordcount_class/ . added manifestadding: home/zyx/wor

24、dcount_class/(in = 0) (out= 0)(stored 0%)adding: home/zyx/wordcount_class/org/(in = 0) (out= 0)(stored 0%)adding: home/zyx/wordcount_class/org/apache/(in = 0) (out= 0)(stored 0%)adding: home/zyx/wordcount_class/org/apache/hadoop/(in = 0) (out= 0)(stored 0%)adding: home/zyx/wordcount_class/org/apache

25、/hadoop/examples/(in = 0) (out= 0)(stored 0%)adding: home/zyx/wordcount_class/org/apache/hadoop/examples/WordCount.class(in = 1911) (out= 996)(deflated 47%)adding: home/zyx/wordcount_class/org/apache/hadoop/examples/WordCount$TokenizerMapper.class(in = 1790) (out= 765)(deflated 57%)adding: home/zyx/

26、wordcount_class/org/apache/hadoop/examples/WordCount$IntSumReducer.class(in = 1789) (out= 746)(deflated 58%)adding: WordCount.class(in = 1911) (out= 996)(deflated 47%)adding: WordCount$TokenizerMapper.class(in = 1790) (out= 765)(deflated 57%)例子：WordCount v1.0在深入細(xì)節(jié)之前，讓我們先看一個(gè)Map/Reduce的應(yīng)用示例，以便對(duì)它們的工作方式

27、有一個(gè)初步的認(rèn)識(shí)。WordCount是一個(gè)簡(jiǎn)單的應(yīng)用，它可以計(jì)算出指定數(shù)據(jù)集中每一個(gè)單詞出現(xiàn)的次數(shù)。這個(gè)應(yīng)用適用于 HYPERLINK /common/docs/r0.18.2/cn/quickstart.html l Standalone+Operation 單 機(jī)模式， HYPERLINK /common/docs/r0.18.2/cn/quickstart.html l SingleNodeSetup 偽 分布式模式 或 HYPERLINK /common/docs/r0.18.2/cn/quickstart.html l Fully-Distributed+Operation 完 全分

28、布式模式 三種Hadoop安裝方式。源代碼WordCount.java1.package org.myorg; 2.3.import java.io.IOException; 4.import java.util.*; 5.6.import org.apache.hadoop.fs.Path; 7.import org.apache.hadoop.conf.*; 8.import org.apache.hadoop.io.*; 9.import org.apache.hadoop.mapred.*; 10.import org.apache.hadoop.util.*; 11.12.publi

29、c class WordCount 13.14. public static class Map extends MapReduceBase implements Mapper 15. private final static IntWritable one = new IntWritable(1); 16. private Text word = new Text(); 17.18. public void map(LongWritable key, Text value, OutputCollector output, Reporter reporter) throws IOExcepti

30、on 19. String line = value.toString(); 20. StringTokenizer tokenizer = new StringTokenizer(line); 21. while (tokenizer.hasMoreTokens() 22. word.set(tokenizer.nextToken(); 23. output.collect(word, one); 24. 25. 26. 27.28. public static class Reduce extends MapReduceBase implements Reducer 29. public

31、void reduce(Text key, Iterator values, OutputCollector output, Reporter reporter) throws IOException 30. int sum = 0; 31. while (values.hasNext() 32. sum += values.next().get(); 33. 34. output.collect(key, new IntWritable(sum); 35. 36. 37.38. public static void main(String args) throws Exception 39.

32、 JobConf conf = new JobConf(WordCount.class); 40. conf.setJobName(wordcount); 41.42. conf.setOutputKeyClass(Text.class); 43. conf.setOutputValueClass(IntWritable.class); 44.45. conf.setMapperClass(Map.class); 46. conf.setCombinerClass(Reduce.class); 47. conf.setReducerClass(Reduce.class); 48.49. con

33、f.setInputFormat(TextInputFormat.class); 50. conf.setOutputFormat(TextOutputFormat.class); 51.52. FileInputFormat.setInputPaths(conf, new Path(args0); 53. FileOutputFormat.setOutputPath(conf, new Path(args1); 54.55. JobClient.runJob(conf); 57. 58. 59.用法假設(shè)環(huán)境變量HADOOP_HOME對(duì)應(yīng)安裝時(shí)的根目錄，HADOOP_VERSION對(duì)應(yīng)Hadoop的當(dāng)前安裝版本，編譯WordCount.java來(lái)創(chuàng)建jar包，可如下操作：$ mkdir wordcount_classes $ javac -classpath $HADOO