數(shù)據(jù)分析平臺hadoop9第六周作業(yè)plugin

Hadoop#9第六周作業(yè)HADOOP1.2.1+ECLIPSEPLUGIN編譯eclipse插件需要的src包，不需要修改集 和解壓hadoop- 并配置Step1導入Hadoop-eclipse插件工獲取hadoop-解壓縮到Eclipse 在Eclipse中選擇File—>Import—>ExistingProjectsintoWorkspace導入已有項F:\workspace\hadoop-1.2.1\src\contrib\eclipse-plugin，其默認的項目名稱是MapReduceTools如圖所示：Step2導入所需jar在項目MapReduceTools中新建lib ，將hadoop-1.2.1下的hadoop-core-1.2.1.jar（重命名為hadoop-core.jar），及其lib 下的commons-cli-1.2.jar、 mons-configuration-1.6.jar、jackson-mapper-asl- mons-http -3.0.1.jar拷貝到該 Step3build-F:\workspace\hadoop-1.2.1\src\contrib\build-contrib.xml（eclipse-plugin）拷貝到MapReduceTools項 中build-contrib.xml<!—修改hadoop存 —<!—添加eclipse存 —<propertyname="eclipse.home"location="F:/eclipse"<!hadoop版本—<propertyname="version"Step4<!—1.修改build-contrib.xml<importfile="build-<!—2.添加如下，否則提示"軟件包org.apache.hadoop.fs不存在"—><pathid="hadoop-<fileset<includename="hadoop-<path<pathelement<pathelement<pathrefid="eclipse-sdk-<pathrefid="hadoop-<!—3.修改并添加jar包拷貝—><targetname="jar"depends="compile"<mkdir<copyfile="${hadoop.root}/hadoop-core-${version}.jar"tofile="${build.dir}/lib/hadoop-core.jar"verbose=""true"/><copyfile="${hadoop.root} mons-cli-1.2.jar"todir="${build.dir}/lib"<copyfile="${hadoop.root} mons-lang-2.4.jar"todir="${build.dir}/lib"<copyfile="${hadoop.root} todir="${build.dir}/lib"verbose="true"/><copyfile="${hadoop.root}/lib/jackson-mapper-asl-1.8.8.jar"todir="${build.dir}/lib"verbose="true"/><copyfile="${hadoop.root}/lib/jackson-core-asl-1.8.8.jar"todir="${build.dir}/lib"verbose="true"/><copyfile="${hadoop.root} todir="${build.dir}/lib"verbose="true"/><jarjarfile="${build.dir}/hadoop-${name}-${version}.jar"<filesetdir="${build.dir}"includes="classes/<filesetdir="${root}"includes="resources/Step5修改META-Bundle-ClassPath:classes/,lib/hadoop-core.jar, Step6Ant編右鍵MapReduceTools工程，Properties—>Builders—>New—>AntBuilderNameBuildfile(build.xmlProject—>Build編譯成功會輸出BUILDSUCCESSFULStep7安裝插編譯結果在F:\workspace\hadoop-1.2.1\build\contrib\eclipse- 將hadoop-eclipse-plugin-1.2.1.jar放入 下，重啟Step8配置插C:\Windows\System32\drivers\etc\hosts28如果不配置，那么在eclipse里面配置ip也可以，必須保證windows可以集群機器hadoop安裝位置（即hadoop-1.2.1解壓位置Map/Reducelocations中新建Hadoop打開map/ReduceMap/ReduceLocationsHadoopLocationView中，右鍵-->NewHadoopLocation。在彈出的框中你需要配置Locationname，如Hadoop，還有Map/ReduceMasterDFSMasterHost、Portmapred-site.xml、core-site.xml中配置的地址及端口。如：切記username一定是集群的用戶名：hadoop，設置Advanced修改hadoop.tmp.dir為你hadoop（core-site.xml）集群中設置的修改dfs.replication為你hdfs-site.xml文件中配置的值重啟eclipseStep9查看是否成eclipsehadoop集群里的EclipsehdfsHDFSeclipsehadoopplugin配置成功。為了簡便其間，只了一少部分的氣象數(shù)據(jù)如下將所有數(shù)據(jù)輸入到一個txt中sudo *.gz>將此文檔提交的HDFS文件系統(tǒng)/opt/hadoop/bin/hadoopfs-查看提交是否成hadoop-core-1.2.1.jarhdfs-site.xml文件（重要windowhadoophdfs-site.xml另個需要修改org.apache.hadoop.fs.FileUtil文件，將其中的privatestaticvoidcheckReturnValue(booleanrv,FileFsPermission)throwsIOExceptionif(!rv)/*thrownewIOException("Failedtosetpermissionsofpath:"p"to"+}}然后將這個源文件重新編譯好后會得到FileUtil$ 兩個類文件，用解壓文件打開hadoop-core-1.1.2_core.jar,然后將這兩個文件替換掉原來運行程：13/10/1319:22:04WARNutil.NativeCodeLoader:Unabletoloadnative-hadooplibraryforyourtform...usingbuiltin-javaclasseswhere13/10/1319:22:04WARNmapred.Job :UseGenericOptionsParserforparsingthearguments.ApplicationsshouldimplementToolforthe13/10/1319:22:04INFOinput.FileInputFormat:Totalinputpathstoprocess:113/10/1319:22:04WARNsnappy.LoadSnappy:Snappynativelibrarynot13/10/1319:22:04INFOmapred.Job :Runningjob: 13/10/1319:22:04INFOmapred.LocalJobRunner:Waitingformaptasks13/10/1319:22:04INFOmapred.LocalJobRunner:Startingtask: 13/10/1319:22:04INFOmapred.Task:UsingResourceCalculatorPlugin:13/10/1319:22:04INFOmapred.MapTask:Processingsplit:13/10/1319:22:04INFOmapred.MapTask:io.sort.mb=10013/10/1319:22:04INFOmapred.MapTask:databuffer=/13/10/1319:22:04INFOmapred.MapTask:recordbuffer=262144/32768013/10/1319:22:04INFOmapred.MapTask:Startingflushofmapoutput13/10/1319:22:04INFOmapred.MapTask:Finishedspill013/10/1319:22:04INFO _0001_m_000000_0isdone.Andisintheprocessofcommiting13/10/1319:22:04INFO13/10/1319:22:04INFOmapred.Task:Task _0001_m_000000_0'done.13/10/1319:22:04INFOmapred.LocalJobRunner:Finishingtask: 13/10/1319:22:04INFOmapred.LocalJobRunner:Maptaskexecutor13/10/1319:22:04INFOmapred.Task:UsingResourceCalculatorPlugin:13/10/1319:22:04INFO13/10/1319:22:04INFOmapred.Merger:Merging1sortedsegments13/10/1319:22:04INFOmapred.Merger:Downtothelastmerge-pass,with1segmentsleftoftotalsize:153023bytes13/10/1319:22:04INFO13/10/1319:22:04INFO _0001_r_000000_0isdone.Andisintheprocessofcommiting13/10/1319:22:04INFO13/10/1319:22:04INFOmapred.Task: _0001_r_000000_0isallowedtocommitnow13/10/1319:22:04INFOou mitter:Savedoutputoftask'attempt_local _0001_r_000000_0'to13/10/1319:22:04INFOmapred.LocalJobRunner:reduce>reduce13/10/1319:22:04INFOmapred.Task:Task _0001_r_000000_0'INFO map100%reduceINFO JobINFO Counters:INFO FileOutputINFO BytesINFO FileInputFormatINFO BytesINFO INFO INFO INFO INFO INFO Map-ReduceINFO Mapoutput13/10/1319:22:05INFOmapred.Job Mapinputrecords=1391413/10/1319:22:05INFOmapred.Job Reduceshufflebytes=013/10/1319:22:05INFOmapred.Job SpilledRecords=2782213/10/1319:22:05INFOmapred.Job Mapoutputbytes=12519913/10/1319:22:05INFOmapred.Job Totalcommittedheapusage(bytes)=13/10/1319:22:05INFOmapred.Job Combineinputrecords=013/10/1319:22:05INFOmapred.Job 13/10/1319:22:05INFOmapred.Job Reduceinput13/10/1319:22:05INFOmapred.Job Reduceinputgroups=113/10/1319:22:05INFOmapred.Job Combineoutputrecords=013/10/1319:22:05INFOmapred.Job Reduceoutputrecords=113/10/1319:22:05INFOmapred.Job Mapoutputrecords=13911查看輸出的結果是通過命令查看的結果是windowsOK錯誤及解問題1：在eclipse第一次運行錯如下13/10/1312:42:38WARNutil.NativeCodeLoader:Unabletoloadnative-hadooplibraryforyourtform...usingbuiltin-javaclasseswhere13/10/1313/10/1312:42:38ERRORsecurity.UserGroupInformation:PriviledgedActionExceptionas:Administratorcause:java.io.IOException:Failedtosetpermissionsof \.stagingto0700Exceptioninthread"main"java.io.IOException:Failedtosetpermissionsofpath:\tmp\hadoop- \.stagingto0700atorg.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:664)atorg.apache.hadoop.mapred.Job atorg.apache.hadoop.mapred.Job atjava.security.AccessController.doPrivileged(NativeMethod)atjavax.security.auth.Subject.doAs(UnknownSource) atorg.apache.hadoop.mapreduce.Job.submit(Job.java:550) atMinTemperature.main(MinTemperature.java:23)eclipse13/10/1312:53:41WARNutil.NativeCodeLoader:Unabletoloadnative-hadooplibraryforyourtform...usingbuiltin-javaclasseswhere13/10/1313/10/1312:53:41ERRORsecurity.UserGroupInformation:PriviledgedActionExceptionas:Administratorcause:java.io.IOException:Failedtosetpermissionsof \.stagingExceptioninthread"main"java.io.IOException:Failedtosetpermissionsofpath: \.stagingtoatorg.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:664)atorg.apache.hadoop.mapred.Job atorg.apache.hadoop.mapred.Job atjava.security.AccessController.doPrivileged(NativeMethod)atjavax.security.auth.Subject.doAs(UnknownSource) atorg.apache.hadoop.mapreduce.Job.submit(Job.java:550) atMinTemperature.main(MinTemperature.java:23) windowseclipse-hadoopeclipse hadoop賬號，同時將該賬號加到管理Failedtosetpermissionsofpathhadoop-core中的源碼(org.apache.hadoop.fs.FileUtil)，將權限檢查(checkReturnValue()方法中的代碼作)jarhadoop-core-更新了jar包后又報錯如下13/10/1319:15:52WARNutil.NativeCodeLoader:Unabletoloadnative-hadooplibraryforyourtform...usingbuiltin-javaclasseswhere13/10/1319:15:52WARNmapred.Job :UseGenericOptionsParserforparsingthearguments.ApplicationsshouldimplementToolforthe13/10/1319:15:52INFOinput.FileInputFormat:Totalinputpathstoprocess:113/10/1319:15:52WARNsnappy.LoadSnappy:Snappynativelibrarynot13/10/1319:15:53INFOmapred.Job :Runningjob: 13/10/1319:15:53WARNmapred.LocalJobRunner:job_local org.apache.hadoop.security.AccessControlException:Permissiondenied:user=Administrator,access=WRITE,atsun.reflect.NativeConstructorAccessorImpl.newInstance0(Nativeatsun.reflect.NativeConstructorAccessorImpl.newInstance(Unknownatjava.lang.reflect.Constructor.newInstance(UnknownSource)atorg.apache.hadoop.hdfs.DFS atorg.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1161) Causedby:org.apache.hadoop.ipc.RemoteException:org.apache.hadoop.security.AccessControlException:Permissiondenied:user=Administrator,access=WRITE,atsun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethod)atjava.lang.reflect.Method.invoke(Method.java:606)atorg.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)atorg.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1432)atorg.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1428)atjava.security.AccessController.doPrivileged(NativeMethod)atjavax.security.auth.Subject.doAs(Subject.java:415)atatorg.apache.hadoop.ipc..call(.java:1113)atorg.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)at$1.mkdirs(UnknownSource)atsun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethod)atsun.reflect.NativeMethodAccessorImpl.invoke(UnknownSource)atsun.reflect.DelegatingMethodAccessorImpl.invoke(Unknownatjava.lang.reflect.Method.invoke(UnknownSource)at$1.mkdirs(Unknown...413/10/1319:15:54INFOmapred.Job:map0%reduce0%13/10/1319:15:54INFOmapred.Job:Jobcomplete: 13/10/1319:15:54INFOmapred.Job:Counters:：windowhadoophdfs-site.xmlwindowsAdministrator進行驗證了。傳遞參數(shù)問2傳遞參數(shù)問題$hadoopjarExercise_1.jarinput_pathoutput_path必須跟附件"resule_2"一致，并且其中hadoop必須由參數(shù)傳遞。1、上傳輸入數(shù)據(jù)Test_12、先測試一下Exercise_1.jar的運打包為week6.jar13/10/1304:57:02INFOmapred.Job Map-Reduce13/10/1304:57:02INFOmapred.Job: Mapoutputmaterializedbytes=21013/10/1304:57:02INFOmapred.Job: Mapinputrecords=613/10/1304:57:02INFOmapred.Job: Reduceshufflebytes=21013/10/1304:57:02INFOmapred.Job: SpilledRecords=1213/10/1304:57:02INFOmapred.Job Mapoutput13/10/1304:57:02INFOmapred.Job: Totalcommittedheapusage(bytes)=13/10/1304:57:02INFOmapred.Job: CPUtimespent(ms)=85013/10/1304:57:02INFOmapred.Job: Combineinputrecords=013/10/1304:57:02INFOmapred.Job: 13/10/1304:57:02INFOmapred.Job: Reduceinputrecords=613/10/1304:57:02INFOmapred.Job: Reduceinputgroups=113/10/1304:57:02INFOmapred.Job Combineoutput13/10/1304:57:02INFOmapred.Job Physicalmemory(bytes)/10/1304:57:02INFOmapred.Job Reduceoutput13/10/1304:57:02INFOmapred.Job: Virtualmemory(bytes)snapshot=13/10/1304:57:02INFOmapred.Job: Mapoutputrecords=6任務開始：2013-10-13任務結束：2013-10-13任務耗時 3、Excrcise1的結果如下4、修改并測試一下Exercise_2.jar的運打包為Exercise_2.jarhadoop@master:/opt/hadoop/myclass$hadoopjarExercise_2.jarinput/Test_2week6out2運行結果如下所示輸出結果如下命令行運行結hadoop@master:/opt/hadoop/myclass$hadoopfs-cat/user/hadoop/week6out2/*Warning:$HADOOP_HOMEisdeprecated. cat:Filedoesnotexist:/user/hadoop/week6out2/_logsExercise_2Hadoopimportimportimportjava.util.Date;importorg.apache.hadoop.conf.Configured;importimportorg.apache.hadoop.io.LongWritable;importorg.apache.hadoop.io.Text;importimportorg.apache.hadoop.mapreduce.Mapper;importimportimportorg.apache.hadoop.mapreduce.lib.output.FileOutputFormat;importorg.apache.hadoop.util.Tool;importpublicclassExercise_2extendsConfiguredimplementsToolenum{,//}MAP privateStringname;publicvoidsetup(Contextcontext{= //} publicvoidmap(LongWritablekey,Textvalue,Contextcontext)throwsIOException,{Stringline= //源數(shù)//String[]lineSplit=line.split("");Stringcaller=lineSplit[0];Stringcallee=lineSplit[1]; Textout=newText(caller); //}catch(java.lang.ArrayIndexOutOfBoundceptione{cont