實驗指導(dǎo)在eclipse中運行mapreduce程序修訂

1、8. 在 Eclipse 中運行“Word Count” MapReduce 程序（1）在 Eclipse 中創(chuàng)建 “WordCount” MapReduce 項目點擊File菜單，選擇 New - Project：選擇Map/Reduce Project，點擊 Next：填寫 Project name為 MyWordCount，點擊 Finish 創(chuàng)建項目。MyWordCount 項目，選擇 New - Class：在 Name 處填寫 WordCountTest。將如下 WordCountTest 的代碼到該 WordCountTest.java 中。import java.io.IOEx

2、ception; import java.util.Iterator;import java.util.StringTokenizer;import.apache.hadoop.conf.Configuration; import.apache.hadoop.fs.Path;import.apache.hadoop.io.Writable; import.apache.hadoop.io.Text;import.apache.hadoop.mapreduce.Job; import.apache.hadoop.mapreduce.Mapper; import.apache.hadoop.map

3、reduce.Reducer;import.apache.hadoop.mapreduce.lib.input.FileInputFormat;importimport.apache.hadoop.mapreduce.lib.output.FileOutputFormat;.apache.hadoop.util.GenericOptionsParser;public class WordCountTest public WordCountTest() public sic void main(String args) throws Exception Configuration conf =

4、new Configuration();String otherArgs = (new GenericOptionsParser(conf, args).getRemainingArgs(); if(otherArgs.length 2) System.err.prln(Usage: wordcount . ); System.exit(2);Job job = Job.getInstance(conf, word count test); job.setJarByClass(WordCountTest.class); job.setMapperClass(WordCountTest.Toke

5、nizerMapper.class);binerClass(WordCountTest.SumReducer.class); job.setReducerClass(WordCountTest.SumReducer.class); job.setOutputKeyClass(Text.class);job.setOutputValueClass(Writable.class);for(i = 0; i otherArgs.length - 1; +i) FileInputFormat.addInputPath(job, neth(otherArgsi);FileOutputFormat.set

6、OutputPath(job, neth(otherArgsotherArgs.length - 1);System.exit(pletion(true)?0:1);public sic classSumReducer extends Reducer privateWritable result = newWritable();publicSumReducer() public void reduce(Text key, Iterable values, Reducer.Context context) throws IOException,sum = 0;erruptedException

7、Writable val;for(Iterator itr = values.iterator(); itr.hasNext(); sum += val.get() val = (Writable)itr.next();（2）將 Hadoop 配置文件添加到“WordCount” MapReduce 項目將 perties中：到 WordCount 項目下的 src 文件夾（/workspace/WordCount/src）og4j 用于程序的輸出，需要 perties 這個配置文件，如果沒有該文件到項目中，運行程序后在 Console 面板中會出現(xiàn)警告提示

8、：log4j:WARNNoappenderscouldbefoundforlogger雖然不影響程序的正確運行的，但程序運行時無法看到任何提示消息（只能看到出錯信息）。完成后，務(wù)必WordCount 選擇 refresh 進行刷新（不會自動刷新，需要手動刷新），可以看到文件結(jié)構(gòu)如下所示：(.apache.hadoop.metrics2.lib.MutableMetricsFactory).log4j:WARN Please initialize the log4j system properly.log4j:WARN Seefor more info.hfutmaster $ cp /hado

9、op-2.5.2/etc/hadoop/perties /workspace/MyWordCount/srcthis.result.set(sum); context.write(key, this.result);public sic class TokenizerMapper extends Mapper private sic finalWritable one = newWritable(1);private Text word = new Text();public TokenizerMapper() publicvoidmap(Objectkey,Textvalu

10、e,Mapper.Context context) throws IOException,erruptedException StringTokenizer itr = new StringTokenizer(value.toString();while(itr.hasMoreTokens() this.word.set(itr.nextToken(); context.write(this.word, one);（3）通過 Eclipse 運行“MyWordCount” MapReduce 項目點擊中的 Run 圖標，或者Project Explorer 中的 WordCountTest.java，選擇Run As - Run on Hadoop，就可以運行 MapReduce 程序了。不過由于沒有指定參數(shù)，運行時會提示 “Usage: wordcount”，需要通過 Eclipse 設(shè)定一下運行參數(shù)。剛創(chuàng)建的 WordCount.java，選擇 Run As - Run Configurations，在此處可以設(shè)置運行時的相關(guān)參數(shù)（ 如果 Java Application 下面沒有 WordCount， 那么需要先雙擊 JavaApplication ）。 切 換 到“ Arguments ”， 在Program arguments 處填寫