dataX安裝使用手冊(cè)

1、DataX3.0使用手冊(cè)目錄一、dataX概述11dataX作用12DataX3.0框架設(shè)計(jì)23DataX3.0插件體系2二、dataX安裝31創(chuàng)建用戶(hù)組及用戶(hù)32環(huán)境變量配置43安裝python4三、dataX使用41生成配置文件模板42配置定時(shí)任務(wù)73常見(jiàn)問(wèn)題及處理8四、dataX插件開(kāi)發(fā)及安裝81.Reader插件開(kāi)發(fā)82.Writter插件開(kāi)發(fā)93.插件部署10一、 dataX概述1 dataX作用DataX 是一個(gè)異構(gòu)數(shù)據(jù)源離線(xiàn)同步工具，致力于實(shí)現(xiàn)包括關(guān)系型數(shù)據(jù)庫(kù)(MySQL、Oracle等)、HDFS、Hive、ODPS、HBase、FTP等各種異構(gòu)數(shù)據(jù)源之間穩(wěn)定高效的數(shù)據(jù)同步功能

2、。為了解決異構(gòu)數(shù)據(jù)源同步問(wèn)題，DataX將復(fù)雜的網(wǎng)狀的同步鏈路變成了星型數(shù)據(jù)鏈路，DataX作為中間傳輸載體負(fù)責(zé)連接各種數(shù)據(jù)源。當(dāng)需要接入一個(gè)新的數(shù)據(jù)源的時(shí)候，只需要將此數(shù)據(jù)源對(duì)接到DataX，便能跟已有的數(shù)據(jù)源做到無(wú)縫數(shù)據(jù)同步。2 DataX3.0框架設(shè)計(jì)DataX本身作為離線(xiàn)數(shù)據(jù)同步框架，采用Framework + plugin架構(gòu)構(gòu)建。將數(shù)據(jù)源讀取和寫(xiě)入抽象成為Reader/Writer插件，納入到整個(gè)同步框架中。Reader：Reader為數(shù)據(jù)采集模塊，負(fù)責(zé)采集數(shù)據(jù)源的數(shù)據(jù)，將數(shù)據(jù)發(fā)送給Framework。Writer： Writer為數(shù)據(jù)寫(xiě)入模塊，負(fù)責(zé)不斷向Framework取數(shù)據(jù)，

3、并將數(shù)據(jù)寫(xiě)入到目的端。Framework：Framework用于連接reader和writer，作為兩者的數(shù)據(jù)傳輸通道，并處理緩沖，流控，并發(fā)，數(shù)據(jù)轉(zhuǎn)換等核心技術(shù)問(wèn)題。3 DataX3.0插件體系類(lèi)型數(shù)據(jù)源Reader(讀)Writer(寫(xiě))RDBMS 關(guān)系型數(shù)據(jù)庫(kù)MySQL Oracle SQL Server PostgreSQL 達(dá)夢(mèng) 通用RDBMS(支持所有關(guān)系型數(shù)據(jù)庫(kù))阿里云數(shù)倉(cāng)數(shù)據(jù)存儲(chǔ)MaxCompute(原ODPS) Analytic DB(原ADS)  OSS 云數(shù)據(jù)庫(kù)Memcache版(原

4、OCS)HiveNoSQL數(shù)據(jù)存儲(chǔ)Table Store(原OTS) Hbase0.94 Hbase1.1 MongoDB無(wú)結(jié)構(gòu)化數(shù)據(jù)存儲(chǔ)TxtFileJsonFile FTP HDFS二、 dataX安裝1 創(chuàng)建用戶(hù)組及用戶(hù)roothmaster-hdfs:/home/ubuntu# groupadd dataxroothmaster-hdfs:/home/ubuntu# useradd -g datax datax -m -d /home/dataxroothmaster-hdfs:/home/ubuntu# passwd dataxEnt

5、er new UNIX password: Retype new UNIX password: passwd: password updated successfullyroothmaster-hdfs:/home/ubuntu# usermod -G adm -a dataxroothmaster-hdfs:/home/ubuntu# su - dataxdataxhmaster-hdfs:$ iduid=1004(datax) gid=1000(ubuntu) groups=1000(ubuntu),4(adm),1002(datax)2 環(huán)境變量配置export JRE_HOME=$JA

6、VA_HOME/jreexport CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/libexport PATH=$JAVA_HOME/binexport LANG=zh_CN.UTF-83 安裝pythondataxhmaster-hdfs:/home/ubuntu$ pythonPython 2.7.6 (default, Jun 22 2015, 17:58:13) GCC 4.8.2 on linux2Type "help", "copyright", "credits" or "licen

7、se" for more information.三、 dataX使用1 生成配置文件模板命令：bin/python datax.py -r YOUR_READER -w YOUR_WRITER案例1：mysqlreader-> mysqlwriterdataxhmaster-hdfs:/datax3/datax/bin$ python datax.py -r mysqlreader -w mysqlwriterDataX (DATAX-OPENSOURCE-1.0), From Alibaba !Copyright (C) 2010-2015, Alibaba Group.

8、All Rights Reserved.Please refer to the mysqlreader document: Please refer to the mysqlwriter document: Please save the following configuration as a json file and use python DATAX_HOME/bin/datax.py JSON_FILE_NAME.json to run the job. "job": "content": "reader": "na

9、me": "mysqlreader", "parameter": "column": , "connection": "jdbcUrl": , "table": , "password": "", "username": "", "where": "" , "writer": "name": "my

10、sqlwriter", "parameter": "column": , "connection": "jdbcUrl": "", "table": , "password": "", "preSql": , "session": , "username": "", "writeMode": "" , &qu

11、ot;setting": "speed": "channel": "" 案例2：jsonfilereader-> hbasewriterdataxhmaster-hdfs:/datax3/datax/bin$ python datax.py -r jsonfilereader -w hbasewriter DataX (DATAX-OPENSOURCE-1.0), From Alibaba !Copyright (C) 2010-2015, Alibaba Group. All Rights Reserved.Plea

12、se refer to the jsonfilereader document: Please refer to the hbasewriter document: Please save the following configuration as a json file and use python DATAX_HOME/bin/datax.py JSON_FILE_NAME.json to run the job. "job": "content": "reader": "name": "jsonf

13、ilereader", "parameter": "column": , "compress": "zip", "encoding": "", "path": , "writer": "name": "hbase11xwriter", "parameter": "column": , "encoding": "&quo

14、t;, "hbaseConfig": "hbase.cluster.distributed": "", "hbase.rootdir": "", "hbase.zookeeper.quorum": "" , "mode": "", "rowkeyColumn": , "table": "", "versionColumn": "

15、index": "", "value": "" , "setting": "speed": "channel": "" 2 配置定時(shí)任務(wù) 創(chuàng)建任務(wù)列表文件dataxhmaster-hdfs:/datax3/datax/task$ vi crontab.tasks.txt 20,30,40 0-23 * * * /usr/bin/python /home/datax/datax3/datax/bin/datax.py /home/da

16、tax/datax3/datax/job/jsonfile2hbase_fullry_log.terminal_log.json >/data/datax/log/datax.crontab.out.log 2>&12.2 crontab加載系統(tǒng)任務(wù)列表文件dataxhmaster-hdfs:/datax3/datax/task$ crontab crontab.tasks.txt dataxhmaster-hdfs:/datax3/datax/task$ crontab -l20,30,40 0-23 * * * /usr/bin/python /home/datax/d

17、atax3/datax/bin/datax.py /home/datax/datax3/datax/job/jsonfile2hbase_fullry_log.terminal_log.json >/data/datax/log/datax.crontab.out.log 2>&13 常見(jiàn)問(wèn)題及處理.1 /bin/sh: 1: java: not found原因：/home/datax/datax3/datax/bin/ datax.py未取到JAVA_HOME環(huán)境變量值四、 dataX插件開(kāi)發(fā)及安裝4.1. Reader插件開(kāi)發(fā)public class Js

18、onFileReader extends Readerpublic static class Job extends Reader.Job Overridepublic void init() / TODO Auto-generated method stubOverridepublic void destroy() / TODO Auto-generated method stubOverridepublic List<Configuration> split(int adviceNumber) / TODO Auto-generated method stubreturn nu

19、ll;public static class Task extends Reader.TaskOverridepublic void init() / TODO Auto-generated method stubOverridepublic void destroy() / TODO Auto-generated method stubOverridepublic void startRead(RecordSender recordSender) / TODO Auto-generated method stub2. Writter插件開(kāi)發(fā)public class JsonFileWrite

20、r extends Writer public static class Job extends Writer.Job Overridepublic void init() / TODO Auto-generated method stubOverridepublic void destroy() / TODO Auto-generated method stubOverridepublic List<Configuration> split(int mandatoryNumber) / TODO Auto-generated method stubreturn null;publ

21、ic static class Task extends Writer.Task Overridepublic void init() / TODO Auto-generated method stubOverridepublic void destroy() / TODO Auto-generated method stubOverridepublic void startWrite(RecordReceiver lineReceiver) / TODO Auto-generated method stub3. 插件部署目錄及文件說(shuō)明dataxhmaster-hdfs:/datax3/dat

22、ax/plugin/reader/jsonfilereader$ lltotal 56drwxr-xr-x 3 datax ubuntu 4096 Jul 6 16:10 ./drwxr-xr-x 21 datax ubuntu 4096 Jun 21 11:05 ./-rw-r-r- 1 datax ubuntu 36348 Jun 22 17:52 jsonfilereader.jardrwxr-xr-x 2 datax ubuntu 4096 May 16 12:46 libs/-rw- 1 datax ubuntu 315 May 16 11:24 plugin.json-rw- 1 datax ubuntu 149 May 15 18:24 plugin_job_template.jsonl jsonfilereader.jar 插件JAR包文件l