gearman開源任務(wù)調(diào)度系統(tǒng)

1、gearman分布式任務(wù)調(diào)度系統(tǒng)方少森百度一、總體介紹gearman是一個分布式任務(wù)分發(fā)調(diào)度框架，支持多語言、并發(fā)的任務(wù)執(zhí)行，支持負(fù)載均衡gearman具有如下特點(diǎn):1、開源2、支持多語言接口： php、perl、python、C 等；3、靈活：不必拘泥與特定的模式，可以靈活使用分布式框架，女map/reduce4、速度快，開銷小5、可嵌入，很輕6、無單點(diǎn)。二、gearman運(yùn)行機(jī)制簡介gearman 包含 3 個基本組件：client、worker 和 job server。client -創(chuàng)建要運(yùn)行的任務(wù)，并提交給job server。job-server -尋找最合適的worker,并提

2、交任務(wù)給worker。worker -接收job server的任務(wù)，執(zhí)行并返回結(jié)果，結(jié)果通過ob server返回給client。從圖中可以看出，client和worker均為gearman提供的api。三、一個簡單的實(shí)例一個基于PHP的實(shí)例，功能是用于反轉(zhuǎn)字符串，client 代碼：! tmt new Gw ar m ：irLCl i 巨nt:Sell ent ad-dEei-vtif I-fint J.li Mlt-Au r 住中巨1 E H巨 11.：. W-!.fl.1!job server收到任務(wù)請求后，會尋找一個能夠運(yùn)行“revers”worker執(zhí)行該任務(wù)。worker上的代碼

3、如下：=?-rpj p如r t pcirgorker ner Ge sqf ：tliiiii urke r3iV u rker aiidS er v er? rkei- midTuin t i ctl i- evmr w e 吧甲乍r 5匚 t i Qnwh.- 1 Jworker ,:Forkfimc 11 old. Tiyr ever e e_ tunc ti on Jj ob Jr dturn strrev(3j db- wurkloadj .上述事例的運(yùn)行時序圖如下:四、三角色的工作流程C i intSiirvrrSriHMIT JOB.SUMMIT JOB RG SUBMIT JOB

4、 HIGH.SllBMn_.HiB HIGH Bt； SUBMIT JQB IJJ+ SUBMIT JOB L0 BG4JOBPRL SI.IIP匚AMXJ汴冊ww coMPr.RrrJOB COMPLETL.lock一含網(wǎng) ASS1LN一次正常的Gearman任務(wù)執(zhí)行流程如上圖所示：worker向Gearman Server注冊自身可以執(zhí)行的功能worker嘗試獲取一個任務(wù)server通告worker暫無任務(wù)worker通告server： “我先睡會，有活干時再叫醒我client向server發(fā)起任務(wù)請求server喚醒可以完成這項(xiàng)工作的worker （可能會喚醒多個woker）worker

5、向server發(fā)起“饑餓請求，嘗試獲得一個任務(wù)server選定一個worker，將該任務(wù)分配下去通告client： “我安排別人處理你的請求了，耐心等待吧worker辛苦工作一段時間后，向server通告干完了server將結(jié)果反饋給用戶說明：1.任務(wù)分類：按優(yōu)先級分：普通（SUBMIT JOB，高（SUBMIT JOB HIGH，低（SUBMIT JOB LOW 按執(zhí)行方式分：普通(_JOB_HIGH,_JOB_LOW，后臺(_JOB_HIGH_BG,_JOB_LOW_BG 最大區(qū)別 在于，client可以跟蹤前臺任務(wù)的工作狀態(tài)，而不能跟蹤BG任務(wù)任務(wù)工作狀態(tài)的通告(worker-serve

6、r-client)：WORK_DATA:WORK_WARNINGWORK_STATUS對于長任務(wù)，worker應(yīng)該每隔一段時間通告任務(wù)狀態(tài)WORK_COMR_ETEWORK_FAILWORK_EXCERTIONServer 監(jiān)控Gearman有AdministrativeProtocol專門用于對Gearman Server的監(jiān)控，主要涉及以下幾方面：status:所注冊職能分類，worker總數(shù)目，處于工作狀態(tài)的worker數(shù)目，可用worker數(shù)目等worker的詳細(xì)信息：所注冊功能、IRserver的緩存任務(wù)最大隊(duì)列長度：可以被查詢也可以被設(shè)定五、通信協(xié)議1、二進(jìn)制packet格式所有請

7、求和返回均為二進(jìn)制數(shù)據(jù)包，包含header和可選的數(shù)據(jù)。header包含如下字段：4 byte magic code - This is either 0REQ for requests or 0RES for responses.4 byte type - A big-endian (network-order) integer containingan enumerated packet type. Possible values are:#NameMagicType1CAN_DOREQWorker2CANT_DOREQWorker3RESET_ABILITIESREQWorker4PRE

8、_SLEEPREQWorker5(unused)-6NOOPRESWorker7SUBMIT_JOBREQClient8JOB_CREATEDRESClient9GRAB_JOBREQWorker10NO_JOBRESWorker11JOB_ASSIGNRESWorker12WORK_STATUSREQWorkerRESClient13WORK_COMPLETEREQWorkerRESClient14WORK_FAILREQWorkerRESClient15GET_STATUSREQClient16ECHO_REQREQClient/Worker17ECHO_RESRESClient/Work

9、er18SUBMIT_JOB_BGREQClient19ERRORRESClient/Worker20STATUS_RESRESClient21SUBMIT_JOB_HIGHREQClient22SET_CLIENT_IDREQWorker23CAN_DO_TIMEOUTREQWorker24ALL_YOURSREQWorker25WORK_EXCEPTIONREQWorkerRESClient26OPTION_REQREQClient/Worker27OPTION_RESRESClient/Worker28WORK_DATAREQWorkerRESClient29WORK_WARNINGRE

10、QWorkerRESClient30GRAB_JOB_UNIQREQWorker31JOB_ASSIGN_UNIQRESWorker32SUBMIT_JOB_HIGH_BGREQClient33SUBMIT_JOB_LOWREQClient34SUBMIT_JOB_LOW_BGREQClient35SUBMIT_JOB_SCHEDREQClient36SUBMIT_JOB_EPOCHREQClient4 byte size - A big-endian (network-order) integer containingthe size of the data being sent after

11、 the header.data域中的參數(shù)之間通過一個字節(jié)的NULL分割，最后一個參數(shù)的界定范圍為NULL分隔符之后到data域總長度之間 的數(shù)據(jù)。參數(shù)的長度不能超過64字節(jié)(包括NULL分隔符)。client/worker 與 job server交互 packet 格式如下:Client/Worker RequestsThese request types may be sent by either a client or a worker:ECHO_REQWhen a job server receives this request, it simply generates a ECHO

12、_RES packet with the data. This is primarily used for testing or debugging.Arguments:-Opaque data that is echoed back in response.Client/Worker ResponsesThese response types may be sent to either a client or a worker:ECHO_RESThis is sent in response to a ECHO_REQ request. The server doesnt look at o

13、r modify the data argument, it just sends it back.Arguments:-Opaque data that is echoed back in response.ERRORThis is sent whenever the server encounters an error and needs to notify a client or worker.Arguments:-NULL byte terminated error code string.-Error text.Client RequestsThese request types m

14、ay only be sent by a client:SUBMIT_JOB, SUBMIT_JOB_BG,SUBMIT_JOB_HIGH, SUBMIT_JOB_HIGH_BG,SUBMIT_JOB_LOW, SUBMIT_JOB_LOW_BGA client issues one of these when a job needs to be run. The server will then assign a job handle and respond with a JOB_CREATED packet.If on of the BG versions is used, the cli

15、ent is not updated with status or notified when the job has completed (it is detached).The Gearman job server queue is implemented with three levels: normal, high, and low. Jobs submitted with one of the HIGH versions always take precedence, and jobs submitted with the normal versions take precedenc

16、e over the LOW versions.Arguments:-NULL byte terminated function name.-NULL byte terminated unique ID.-Opaque data that is given to the function as an argument.SUBMIT_JOB_SCHEDJust like SUBMIT_JOB_BG, but run job at given time instead of immediately. This is not currently used and may be removed.Arg

17、uments:-NULL byte terminated function name.-NULL byte terminated unique ID.-NULL byte terminated minute (0-59).-NULL byte terminated hour (0-23).-NULL byte terminated day of month (1-31).-NULL byte terminated month (1-12).-NULL byte terminated day of week (0-6, 0 = Monday).-Opaque data that is given

18、 to the function as an argument.SUBMIT_JOB_EPOCHJust like SUBMIT_JOB_BG, but run job at given time instead of immediately. This is not currently used and may be removed.Arguments:-NULL byte terminated function name.-NULL byte terminated unique ID.-NULL byte terminated epoch time.-Opaque data that is

19、 given to the function as an argument.GET_STATUSA client issues this to get status information for a submitted job.Arguments:-Job handle that was given in JOB_CREATED packet.OPTION_REQA client issues this to set an option for the connection in the job server. Returns a OPTION_RES packet on success,

20、or an ERROR packet on failure.Arguments:-Name of the option to set. Possibilities are:* exceptions - Forward WORK_EXCEPTION packets to the client.Client ResponsesThese response types may only be sent to a client:JOB_CREATEDThis is sent in response to one of the SUBMIT_JOB* packets. It signifies to t

21、he client that a the server successfully received the job and queued it to be run by a worker.Arguments:-Job handle assigned by server.WORK_DATA, WORK_WARNING, WORK_STATUS, WORK_COMPLETE, WORK_FAIL, WORK_EXCEPTIONFor non-background jobs, the server forwards these packets from the worker to clients.

22、See Worker Requests for more information and arguments.STATUS_RESThis is sent in response to a GET_STATUS request. This is used by clients that have submitted a job with SUBMIT_JOB_BG to see if the job has been completed, and if not, to get the percentage complete.Arguments:-NULL byte terminated job

23、 handle.-NULL byte terminated known status, this is 0 (false) or 1 (true).-NULL byte terminated running status, this is 0 (false) or 1 (true).-NULL byte terminated percent complete numerator.-Percent complete denominator.OPTION_RESSuccessful response to the OPTION_REQ request.Arguments:-Name of the

24、option that was set, see OPTION_REQ for possibilities.Worker RequestsThese request types may only be sent by a worker:CAN_DOThis is sent to notify the server that the worker is able to perform the given function. The worker is then put on a list to be woken up whenever the job server receives a job

25、for that function.Arguments:-Function name.CAN_DO_TIMEOUTSame as CAN_DO, but with a timeout value on how long the job is allowed to run. After the timeout value, the job server will mark the job as failed and notify any listening clients.Arguments:-NULL byte terminated Function name.-Timeout value.C

26、ANT_DOThis is sent to notify the server that the worker is no longerable to perform the given function.Arguments:-Function name.RESET_ABILITIESThis is sent to notify the server that the worker is no longer able to do any functions it previously registered with CAN_DO or CAN_DO_TIMEOUT.Arguments:-Non

27、e.PRE_SLEEPThis is sent to notify the server that the worker is about to sleep, and that it should be woken up with a NOOP packet if a job comes in for a function the worker is able to perform.Arguments:-None.GRAB_JOBThis is sent to the server to request any available jobs on the queue. The server w

28、ill respond with either NO_JOB or JOB_ASSIGN, depending on whether a job is available.Arguments:-None.GRAB_JOB_UNIQJust like GRAB_JOB, but return JOB_ASSIGN_UNIQ when there is a job.Arguments:-None.WORK_DATAThis is sent to update the client with data from a running job. A worker should use this when

29、 it needs to send updates, send partial results, or flush data during long running jobs. It can also beused to break up a result so the worker does not need to buffer the entire result before sending in a WORK_COMPLETE packet.Arguments:-NULL byte terminated job handle.-Opaque data that is returned t

30、o the client.WORK_WARNINGThis is sent to update the client with a warning. It acts just like a WORK_DATA response, but should be treated as a warning instead of normal response data.Arguments:-NULL byte terminated job handle.-Opaque data that is returned to the client.WORK_STATUSThis is sent to upda

31、te the server (and any listening clients) of the status of a running job. The worker should send these periodically for long running jobs to update the percentage complete. The job server should store this information so a client who issued a background command may retrieve it later with a GET_STATU

32、S request.Arguments:-NULL byte terminated job handle.-NULL byte terminated percent complete numerator.-Percent complete denominator.WORK_COMPLETEThis is to notify the server (and any listening clients) that the job completed successfully.Arguments:-NULL byte terminated job handle.-Opaque data that i

33、s returned to the client as a response.WORK_FAILThis is to notify the server (and any listening clients) that the job failed.Arguments:-Job handle.WORK_EXCEPTIONThis is to notify the server (and any listening clients) that the job failed with the given exception.Arguments:-NULL byte terminated job h

34、andle.-Opaque data that is returned to the client as an exception.SET_CLIENT_IDThis sets the worker ID in a job server so monitoring and reporting commands can uniquely identify the various workers, and different connections to job servers from the same worker.Arguments:-Unique string to identify th

35、e worker instance.ALL_YOURSNot yet implemented. This looks like it is used to notify a job server that this is the only job server it is connected to, so a job can be given directly to this worker with a JOB_ASSIGN and no worker wake-up is required.Arguments:-None.Worker ResponsesThese response type

36、s may only be sent to a worker:NOOPThis is used to wake up a sleeping worker so that it may grab a pending job.Arguments:-None.NO_JOBThis is given in response to a GRAB_JOB request to notify the worker there are no pending jobs that need to run.Arguments: -None.JOB_ASSIGNThis is given in response to

37、 a GRAB_JOB request to give the worker information needed to run the job. All communication about the job (such as status updates and completion response) should use the handle, and the worker should run the given function with the argument.Arguments:-NULL byte terminated job handle.-NULL byte termi

38、nated function name.-Opaque data that is given to the function as an argument.JOB_ASSIGN_UNIQThis is given in response to a GRAB_JOB_UNIQ request and acts just like JOB_ASSIGN but with the client assigned unique ID.Arguments:-NULL byte terminated job handle.-NULL byte terminated function name.-NULL

39、byte terminated unique ID.-Opaque data that is given to the function as an argument.2、管理協(xié)議job server支持文本的命令，用于獲取信息和執(zhí)行管理任務(wù)。文本命令與二進(jìn)制命令使用相同的端口，二者之間通過命 令的第一個字節(jié)進(jìn)行區(qū)分：若第一個字節(jié)為NULL則為二進(jìn)制命令，非NULL則為文本命令。管理命令如下：workersThis sends back a list of all workers, their file descriptors,their IPs, their IDs, and a list

40、of registered functions they canperform. The list is terminated with a line containing a single.(period). The format is:FD IP-ADDRESS CLIENT-ID : FUNCTION .Arguments:-None.statusThis sends back a list of all registered functions. Next to each function is the number of jobs in the queue, the number o

41、f running jobs, and the number of capable workers. The columns are tab separated, and the list is terminated Wth a line containing a single . (period). The format is:FUNCTIONtTOTALtRUNNINGtAVAILABL_WORKERSArguments:- None.maxqueueThis sets the maximum queue size for a function. If no size isgiven, t

42、he default is used. If the size is negative, then the queue is set to be unlimited. This sends back a single line Wh OK.Arguments:-Function name.-Optional maximum queue size.shutdownShutdown the server. If the optional graceful argument is used, close the listening socket and let all existing connec

43、tions complete.Arguments:-Optional graceful mode.versionSend back the version of the server.Arguments:-None.六、上面的reverse實(shí)例socket交互分析Worker registration:Worker - Job Server00 52 45 510REQ(Magic)00 00 00 011(Packet type: CAN_DO)00 00 00 077(Packet length)72 65 76 65 72 73 65reverse(Function)Worker check for job:Worker - Job Server00 52 45 510REQ(Magic)00 00 00 099(Packet type: GRAB_JOB)00 00