hadoop搭建與eclipse開發(fā)環(huán)境設置

hadoophadoop開建也是VMWare6.5虛擬機上進行Hadoop版本:hadoop-0.20.2Eclipse版本:3.3.2Java版本：1.6.0_26（1.6以上3.3.2版本）ubuntuhadoop-0.20.203環(huán)境一直沒有成功。但是開發(fā)測試程序如果要減少權限問題的發(fā)生，可以這樣做：ubuntuhadoophadoop-0.20.2時使用hadoop用開發(fā)測試環(huán)境Ubuntuhadoop也安裝配置好了，再克隆二份，然后把主機名與IP修改，再進行主機之間的SSH配置。要的時候通過apt-getinstall進行安裝。不過SSH服務是必須的。Hadoop$ $ $ exportJAVA_HOME=/opt/jdk1.6.0_26exportJAVA_HOME=/opt/jdk1.6.0_26exportPATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATHexportHADOOP_HOME=/opt/hadoop export加入export<?xml<?xml<!--Putsite-specificpropertyoverridesinthisfile.--<description>Abaseforothertemporary<description>Thenameofthedefaultfilesystem.AURIwhoseschemeandauthoritydeterminetheFileSystemimplementation.Theuri'sschemedeterminestheconfigproperty(fs.SCHEME.impl)namingtheFileSystemimplementationclass.Theuri'sauthorityisusedtodeterminethehost,port,etc.forafilesystem.</description><description>DetermineswhereonthelocalfilesystemtheDFSnamenodeshouldstorethenametable.Ifthisisacomma-delimitedlistofdirectoriesthenthenametableisreplicatedinallofthedirectories,forredundancy.</description><?xml<!--<?xml<!--Putsite-specificpropertyoverridesinthisfile.--<description>DetermineswhereonthelocalfilesystemanDFSdatanodeshouldstoreitsblocks.Ifthisisacomma-delimitedlistofdirectories,thendatawillbestoredinallnameddirectories,typicallyondifferentdevices.Directoriesthatdonotexistareignored.</description><description>Defaultblockreplication.Theactualnumberofreplicationscanbespecifiedwhenthefileiscreated.Thedefaultisusedifreplicationisnotspecifiedincreate<?xml<?xml<!--Putsite-specificpropertyoverridesinthisfile.--<description>ThehostandportthattheMapReducejobtrackerrunsat.If"local",thenjobsarerunin-processasasinglemapandreducetask.</description>改相應的主機名和IP地址，這樣就可以簡單地保持hadoop環(huán)境基本配置相同。hadoop安裝目錄拷貝到ubuntu1和ubuntu2相應的目錄。$$mkdir$$mkdir$ssh-keygen–t$ssh-keygen–t$ssh$$ssh$cpid_rsa.pub$ssh$sshubuntuubuntu1$ssh$ssh首先說明，hadoop命令和參數都是大小寫敏感的，該用大寫時用大寫，用小寫時用小$$hadoopnamenode-$start-$start-$stop-$stop-$$2971297130432857422931542737$1005$100522751090 訪問 訪問 可以查看NameNode及整個分布式文件系統(tǒng)的狀態(tài)等$cd$hadoopfs$cd$hadoopfs-mkdir$hadoopjarhadoop-0.20.2-examples.jarwordcountinput$hadoopfscat 安裝包需要注意的是插件版本（jar包）hadoop一致，否則可能會出現EOFException異常。到map/reduce開發(fā)視圖。后彈出的對話框（Generaltab）進行參數的添加：:：與core-site.xml里設置一致。dfs.replication：與hdfs-site.xml里面的dfs.replication一致。hadoop.tmp.dir：與core-site.xml里hadoop.tmp.dir設置一致。hadoop.job.ugi：并不是設置用戶名與密碼。是用戶與組名，所以這里填寫ProjectExplorerDFSLocations目錄，應該可以年看到文件系統(tǒng)中的結構了。但是在/hadoop/mapred/system下卻沒有查看權限，如下圖：的）進行遠程hadoop系統(tǒng)操作，沒有權限。了，將hadoop加在第一個，并用逗號分隔。如：文件內容讀為字符串，將字符串寫入文件。同時有個main函數，可以修改測試:packagepackageimportjava.io.BufferedReader;importjava.io.IOException;importjava.io.InputStream;importorg.apache.hadoop.conf.Configuration;importorg.apache.hadoop.fs.FileSystem;import*Theutilitiestooperatefileonhadoop*@authorluolihui2011-07-publicclassDFSOperatorprivatestaticfinalintBUFFER_SIZE=4096;*publicCreateafileonhdfs.Therootpathis@param thefilenameto@paramoverwriteifafilewiththisnamealreadyexists,theniftrue,thefilewill@returntrueifdeleteissuccessfulelse{//Stringuri=FileSystemfs=FileSystem.get(conf);Pathf=newPath(ROOT_PATH+path);fs.create(f,overwrite);}Deleteafileonhdfs.Therootpathis/.@parampaththepathto@paramrecursiveifpathisadirectoryandsettotrue,thedirectoryisdeletedelsethrowsanexception.Incaseofafiletherecursivecanbesettoeithertrueorfalse.@returntrueifdeleteissuccessfulelsepublicstaticbooleandeleteFile(Stringpath,booleanrecursive)throws{//Stringuri=FileSystemfs=FileSystem.get(conf);Pathf=newPath(ROOT_PATH+path);fs.delete(f,recursive);}Readafiletostringonhadoophdfs.Fromstreamtostring. @parampaththepathto@returntrueifreadissuccessfulelse{FileSystemfs=FileSystem.get(conf);Pathf=newPath(ROOT_PATH+path);InputStreamin=null;Stringstr=StringBuildersb=newStringBuilder(BUFFER_SIZE);if(fs.exists(f)){in=BufferedReaderbf=newBufferedReader(newwhile((str=bf.readLine())!={}}{}

return}Writestringtoahadoophdfsfile.forforexample:DFSOperator.writeStringToDFSFile("/lory/test1.txt","Youarea@parampaththefilewherethestringtowrite@paramstringthecontexttowriteina@returntrueifwriteissuccessfulelse{FileSystemfs=FileSystem.get(conf);FSDataOutputStreamos=null;os=fs.create(f,true);}publicstaticvoidmain(String[]{tryDFSOperator.createFile("/lory/test1.txt",true);DFSOperator.deleteFile("/dfs_operator.txt",true); }catch(IOExceptione)}}}RunAsRunonHadoopChooseanexitsingserverfromthelist結果很簡單（那個警告不管foundintheclasspath.Usageofhadoop-site.xmlisdeprecated.Insteadusecore-site.xml,mapred-site.xmlandhdfs-site.xmltooverrideYouareabadman.參數，然后同樣“Runonhadoop”。具體步驟不再示范。jarRunonhadoop，以后只需要點擊那運行的綠色按鈕了。類似的運行hadoop程序時，有時候會報以下錯誤：Namenodeisinsafebin/hadoopdfsadmin-safemode用戶可以通過dfsadmin-safemode enter-leaveNameNode離開安全模式get-返回安全模式是否開啟的信息wait-等待，一直到安全模式結束。org.apache.hadoop.security.AccessControlException:Permissiondenied:user=Administrator,access=WRITE,atorg.apache.hadoop.fs.FileSystem.create(FileSystem.java:484)atorg.apache.hadoop.fs.FileSystem.create(FileSystem.java:465)atorg.apache.hadoop.fs.FileSystem.create(FileSystem.