認(rèn)證數(shù)據(jù)工程師Flume實驗

Flume實驗1.TDH4.8以前（包括TDH4.8）使用yum方式從本地源安裝flume#yuminstallflumeTDH4.8.1開始通過flume官方網(wǎng)站下載flume安裝包，放到/mnt/disk1下。2.配置環(huán)境變量修改/etc/profile，輸入：vim/etc/profile加上參數(shù):exportJAVA_HOME=/usr/java/jdk1.7.0_71exportHADOOP_HOME=/usr/lib/hadoopexportHDFS_HOME=/usr/lib/hadoop-hdfsexportHADOOP_CLASSPATH=$HADOOP_HOME:$HADOOP_HOME/lib:$HDFS_HOME/lib3.創(chuàng)建flume-env.sh文件。進(jìn)入flume目錄：cd/$FLUME_HOME/flume/conf創(chuàng)建：vimflume-env.sh內(nèi)容如下：FLUME_HOME="/usr/lib/flume"FLUME_CLASSPATH="$FLUME_HOME/lib"JAVA_HOME="/usr/java/jdk1.7.0_71"JAVA_OPTS="-Xms512m-Xmx1024m-Dcom.sun.management.jmxremote"HADOOP_HOME="/usr/lib/haoop"HDFS_HOME="/usr/lib/hdfs"HADOOP_CLASSPATH="$HADOOP_HOME:$HADOOP_HOME/lib:$HDFS_HOME/lib"4.在/$FLUME_HOME/conf目錄下編輯flume.conf配置文件，設(shè)置source類型為分別為目錄、文件，sink類型為hdfsflume.conf內(nèi)容：#a.conf:Asingle-nodeFlumeconfiguration#Namethecomponentsonthisagenta1.sources=r1a1.sinks=k1a1.channels=c1#Describe/configurethesourcea1.sources.r1.type=spooldira1.sources.r1.spoolDir=/opt/vdb/loga1.sources.r1.fileHeader=truea1.sources.r1.deserializer.outputCharset=UTF-8#Describethesinka1.sinks.k1.type=hdfsa1.sinks.k1.hdfs.path=hdfs://tdh-101:8020/tmp/bxu/file1a1.sinks.k1.hdfs.fileType=DataStreama1.sinks.k1.hdfs.writeFormat=Texta1.sinks.k1.hdfs.maxOpenFiles=1a1.sinks.k1.hdfs.rollCount=0a1.sinks.k1.hdfs.rollInterval=0a1.sinks.k1.hdfs.rollSize=1000000a1.sinks.k1.hdfs.batchSize=10000#Useachannelwhichbufferseventsinmemorya1.channels.c1.type=memorya1.channels.c1.capacity=1000000a1.channels.c1.transactionCapacity=10000#Bindthesourceandsinktothechannela1.sources.r1.channels=c1a1.sinks.k1.channel=c15.在本地創(chuàng)建flume監(jiān)控的文件目錄或者文件，并設(shè)置該目錄或者文件的權(quán)限為777操作命令記錄：如上圖所示的demo，上圖中的紅色部分目錄需要手動創(chuàng)建。#mkdir-r/opt/vdb/log/#chmod777-R/opt/vdb6.執(zhí)行以下命令打開flume的常駐進(jìn)程，對指定的文件目錄或者文件進(jìn)行監(jiān)控#cd/usr/lib/flume#bin/flume-ngagent--confconf--conf-fileconf/flume.conf--namea1-Dflume.root.logger=INFO,console以上內(nèi)容為，監(jiān)控文件夾，如果linux目錄的文件夾下，有文件添加，那么會自動采集到HDFS目錄，如果需要監(jiān)控具體的文件內(nèi)容，如果該文件中有數(shù)據(jù)更新，那么需要修改flume.conf文件為如下，其他不變！#a.conf:Asingle-nodeFlumeconfiguration#Namethecomponentsonthisagenta1.sources=r1a1.sinks=k1a1.channels=c1#Describe/configurethesourcea1.sources.r1.type=execmand=tail-F/opt/vdb/log/exec.texta1.sources.r1.fileHeader=truea1.sources.r1.deserializer.outputCharset=UTF-8#Describethesinka1.sinks.k1.type=hdfsa1.sinks.k1.hdfs.path=hdfs://tdh-101:8020/tmp/bxu/file1a1.sinks.k1.hdfs.fileType=DataStreama1.sinks.k1.hdfs.writeFormat=Texta1.sinks.k1.hdfs.maxOpenFiles=1a1.sinks.k1.hdfs.rollCount=0a1.sinks.k1.hdfs.rollInterval=0a1.sinks.k1.hdfs.rollSize=1000000a1.sinks.k1.hdfs.batchSize=10000#Useachannelwhichbufferseventsinmemorya1.channels.c1.type=memorya1.channels.c1.capacity=1000000a1.channels.c1.transactionCapacity=10000#Bindthesourceandsinktothechannela1.sources.r1.cha