2024大數(shù)據(jù)技術(shù)培訓(xùn)Zookeeper

大數(shù)據(jù)技術(shù)培訓(xùn)Zookeeper第1章Zookeeper概述概述Zookeeper是一個(gè)開源的分布式的，為分布式應(yīng)用提供協(xié)調(diào)服務(wù)的Apache項(xiàng)目。ZookeeperZookeeperMaster/Slave管理模式Zookeeper=文件系統(tǒng)+通知機(jī)制特點(diǎn)1）Zookeeper：一個(gè)領(lǐng)導(dǎo)者（leader），多個(gè)跟隨者（follower）組成的集群。2）Leader負(fù)責(zé)進(jìn)行投票的發(fā)起和決議，更新系統(tǒng)狀態(tài)Follower過程中參與投票Zookeeperserverclientserverclient能讀到最新數(shù)據(jù)。數(shù)據(jù)結(jié)構(gòu)ZooKeeper數(shù)據(jù)模型的結(jié)構(gòu)與Unix文件系統(tǒng)很類似，整體上可以看作是一棵樹，每個(gè)節(jié)點(diǎn)稱做一個(gè)ZNode。zookeeperznode1MBZNode應(yīng)用場景

數(shù)據(jù)結(jié)構(gòu)圖典型應(yīng)用場景–數(shù)據(jù)發(fā)布與訂閱集中式配置中心（推+拉）

Zookeeper數(shù)據(jù)模型Server01/ConfigSyslogServer02Database配置管理員應(yīng)用啟動(dòng)時(shí)主動(dòng)到Server01/ConfigSyslogServer02Database配置管理員配置管理員變更Zookeeper配置節(jié)點(diǎn)的內(nèi)容。Zookeeper推送變更到應(yīng)用，觸發(fā)Watcher回調(diào)函數(shù)。應(yīng)用根據(jù)邏輯，主動(dòng)獲取新的配置信息，更改自身邏輯。適用于配置信息多設(shè)備共享，會(huì)發(fā)生動(dòng)態(tài)變化典型應(yīng)用場景–軟負(fù)載均衡文章查詢服務(wù)價(jià)格查詢服務(wù)文章查詢服務(wù)價(jià)格查詢服務(wù)認(rèn)證與授權(quán)服務(wù)Register負(fù)責(zé)域名的注冊，服務(wù)啟動(dòng)后將域名信息通過Register注冊到Zookeeper相對應(yīng)的域名服務(wù)下。Dispatcher負(fù)責(zé)域名的解析?？梢詫?shí)現(xiàn)軟負(fù)載均衡。Register Dispatcher ScannerRegister Dispatcher ScannerSDK（OpenAPI）服務(wù)提供者...服務(wù)消費(fèi)者M(jìn)onitorControllerMonitor負(fù)責(zé)收集服務(wù)信息與狀態(tài)監(jiān)控。Controller提供后臺(tái)Console，提供配置管理功能。典型應(yīng)用場景–集群管理有多少機(jī)器在工作？每臺(tái)機(jī)器的運(yùn)行狀態(tài)收集。對集群中設(shè)備進(jìn)行上下線操作。分布式任務(wù)的狀態(tài)匯報(bào)…下載地址1）官網(wǎng)首頁：/2）下載截圖第2章Zookeeper安裝本地模式安裝部署filezillazookeeperlinux系統(tǒng)下[atguigu@hadoop102software]$tar-zxvfzookeeper-3.4.10.tar.gz-C/opt/module/2）配置修改將/opt/module/zookeeper-3.4.10/conf這個(gè)路徑下的zoo_sample.cfg修改為zoo.cfg；進(jìn)入zoo.cfg文件：vimzoo.cfg修改dataDir路徑為dataDir=/opt/module/zookeeper-3.4.10/zkData在/opt/module/zookeeper-3.4.10/這個(gè)目錄上創(chuàng)建zkData文件夾mkdirzkData3）操作zookeeper啟動(dòng)zookeeper[atguigu@hadoop102zookeeper-3.4.10]$bin/zkServer.shstart[atguigu@hadoop102zookeeper-3.4.10]$jps4020Jps4001QuorumPeerMain[atguigu@hadoop102zookeeper-3.4.10]$bin/zkServer.shstatusZooKeeperJMXenabledbydefaultUsingconfig:/opt/module/zookeeper-3.4.10/bin/../conf/zoo.cfgMode:standalone[atguigu@hadoop102zookeeper-3.4.10]$bin/zkCli.sh[zk:localhost:2181(CONNECTED)0]quit停止zookeeper[atguigu@hadoop102zookeeper-3.4.10]$bin/zkServer.shstop配置參數(shù)解讀解讀zoo.cfg文件中參數(shù)含義tickTime：通信心跳數(shù)，ZookeeperZookeeper使用的基本時(shí)間，服務(wù)器之間或客戶端與服務(wù)器之間維持心跳的時(shí)間間隔，也就是每個(gè)tickTime時(shí)間就會(huì)發(fā)送一個(gè)心跳，時(shí)間單位為毫秒。它用于心跳機(jī)制，并且設(shè)置最小的session超時(shí)時(shí)間為兩倍心跳時(shí)間。(session的最小超時(shí)時(shí)間是2*tickTime)initLimit：LF初始通信時(shí)限集群中的follower跟隨者服務(wù)器(F)與leader領(lǐng)導(dǎo)者服務(wù)器(L)之間初始連接時(shí)能容忍的最多心跳數(shù)（tickTime的數(shù)量），用它來限定集群中的Zookeeper服務(wù)器連接到Leader的時(shí)限。投票選舉新leader的初始化時(shí)間Follower在啟動(dòng)過程中，會(huì)從Leader同步所有最新數(shù)據(jù)，然后確定自己能夠?qū)ν夥?wù)的起始狀態(tài)。Leader允許F在initLimit時(shí)間內(nèi)完成這個(gè)工作。3）syncLimit：LF同步通信時(shí)限集群中Leader與Follower之間的最大響應(yīng)時(shí)間單位，假如響應(yīng)超過syncLimit*tickTime，Leader認(rèn)為Follwer死掉，從服務(wù)器列表中刪除Follwer。在運(yùn)行過程中，Leader負(fù)責(zé)與ZK集群中所有機(jī)器進(jìn)行通信，例如通過一些心跳檢測機(jī)制，來檢測機(jī)器的存活狀態(tài)。如果L發(fā)出心跳包在syncLimit之后，還沒有從F那收到響應(yīng)，那么就認(rèn)為這個(gè)F已經(jīng)不在線了。dataDir保存內(nèi)存數(shù)據(jù)庫快照信息的位置，如果沒有其他說明，更新的事務(wù)日志也保存到數(shù)據(jù)庫。clientPort第3章Zookeeper內(nèi)部原理選舉機(jī)制半數(shù)機(jī)制（Paxos協(xié)議）：集群中半數(shù)以上機(jī)器存活，集群可用。所以zookeeper適合裝在奇數(shù)臺(tái)機(jī)器上。Zookeeperslave。但是，zookeeper工作時(shí)，leaderfollower，LeaderZookeeperServiceServer1myid=1Server2myid=2LeaderServer3 Server4myid=3 myid=4Server5myid=5ZookeeperServiceServer1myid=1Server2myid=2LeaderServer3 Server4myid=3 myid=4Server5myid=5ClientClientClientClientClientClientClientClient1LOOKING狀態(tài)。21id2以上的服務(wù)器都同意選舉它(這個(gè)例子中的半數(shù)以上是3)，所以服務(wù)器1、2還是繼續(xù)保持LOOKING331、、3leader。44234中最了。54節(jié)點(diǎn)類型Znode有兩種類型：短暫（ephemeral）：客戶端和服務(wù)器端斷開連接后，創(chuàng)建的節(jié)點(diǎn)自己刪除持久（persistent）：客戶端和服務(wù)器端斷開連接后，創(chuàng)建的節(jié)點(diǎn)不刪除Znode（persistent）（PERSISTENT）客戶端與zookeeper斷開連接后，該節(jié)點(diǎn)依舊存在（PERSISTENT_SEQUENTIAL）客戶端與zookeeper斷開連接后，該節(jié)點(diǎn)依舊存在，只是Zookeeper給該節(jié)點(diǎn)名稱進(jìn)行順序編號（EPHEMERAL）客戶端與zookeeper斷開連接后，該節(jié)點(diǎn)被刪除（EPHEMERAL_SEQUENTIAL）客戶端與zookeeper斷開連接后，該節(jié)點(diǎn)被刪除，只是Zookeeper給該節(jié)點(diǎn)名稱進(jìn)行順序編號znodestat結(jié)構(gòu)體1）czxid-引起這個(gè)znode創(chuàng)建的zxid，創(chuàng)建節(jié)點(diǎn)的事務(wù)的zxidzxidID。IDzxidzxid1小zxid2，那么zxid1zxid2之前發(fā)生。2）ctime-znode(1970)3）mzxidznodezxid4）mtime-znode最后修改的毫秒數(shù)(從1970年開始)5）pZxid-znode最后更新的子節(jié)點(diǎn)zxidcversionznode子節(jié)點(diǎn)修改次數(shù)dataversionznode數(shù)據(jù)變化號znodeephemeralOwner-sessionid。如果不是臨時(shí)節(jié)0。dataLength-znode的數(shù)據(jù)長度numChildrenznode子節(jié)點(diǎn)數(shù)量監(jiān)聽器原理監(jiān)聽器原理注冊的監(jiān)聽器列表port5“/”路徑數(shù)據(jù)發(fā)生變化注冊的監(jiān)聽器列表port5“/”路徑數(shù)據(jù)發(fā)生變化3getChildren(“/”,true)4Client:ip:port:/path2創(chuàng)建zkClientListener6process()connect

zookeeper線程ZookeeperconnectZookeeper。Zookeeper線程。6）listenerprocess（）方法。getpath[watch]lspath[watch]寫數(shù)據(jù)流程ZooKeeper的寫數(shù)據(jù)流程主要分為以下幾步：比如向ZooKeeper的Server1Server1Server1為每個(gè)的里面有一個(gè)是Leader。這個(gè)會(huì)將寫請求廣播給各個(gè)ServerServer1Server2。收到大多數(shù)三個(gè)節(jié)點(diǎn)的話，只要有兩個(gè)節(jié)點(diǎn)數(shù)據(jù)寫成功了，那么就認(rèn)為數(shù)據(jù)寫成功了。寫成功之后，LeaderServer1數(shù)據(jù)寫成功了。Server1會(huì)進(jìn)一步通知第4章Zookeeper實(shí)戰(zhàn)分布式安裝部署集群規(guī)劃hadoop102、hadoop103hadoop104Zookeeper。1）解壓安裝解壓zookeeper安裝包到/opt/module/目錄下[atguigu@hadoop102software]$tar-zxvfzookeeper-3.4.10.tar.gz-C/opt/module/在/opt/module/zookeeper-3.4.10/這個(gè)目錄下創(chuàng)建zkDatamkdir-pzkData重命名/opt/module/zookeeper-3.4.10/confzoo_sample.cfgzoo.cfgzoo_sample.cfgzoo.cfg配置zoo.cfg文件#######################cluster##########################Server.A=B:C:D。A是一個(gè)數(shù)字，表示這個(gè)是第幾號服務(wù)器；B是這個(gè)服務(wù)器的ip地址；C是這個(gè)服務(wù)器與集群中的Leader服務(wù)器交換信息的端口；D是萬一集群中的Leader服務(wù)器掛了，需要一個(gè)端口來重新進(jìn)行選舉，選出一個(gè)新的Leader，而這個(gè)端口就是用來執(zhí)行選舉時(shí)服務(wù)器相互通信的端口。集群模式下配置一個(gè)文件myid，這個(gè)文件在dataDir目錄下，這個(gè)文件里面有一個(gè)數(shù)據(jù)就是A的值，Zookeeper啟動(dòng)時(shí)讀取此文件，拿到里面的數(shù)據(jù)與zoo.cfg里面的配置信息比較從而判斷到底是哪個(gè)server。3）集群操作在/opt/module/zookeeper-3.4.10/zkDatamyid的文件touchmyid添加myid文件，注意一定要在linux里面創(chuàng)建，在notepad++里面很可能亂碼myid文件vimyid在文件中添加與server對應(yīng)的編號：如2拷貝配置好的zookeeper到其他機(jī)器上scp-rzookeeper-3.4.10/root@:/opt/app/scp-rzookeeper-3.4.10/root@:/opt/app/并分別修改myid文件中內(nèi)容為3、4zookeeper[root@hadoop102zookeeper-3.4.10]#bin/zkServer.shstart[root@hadoop103zookeeper-3.4.10]#bin/zkServer.shstart[root@hadoop104zookeeper-3.4.10]#bin/zkServer.shstart[root@hadoop102zookeeper-3.4.10]#bin/zkServer.shstatusJMXenabledbydefaultUsingconfig:/opt/module/zookeeper-3.4.10/bin/../conf/zoo.cfgMode:follower[root@hadoop103zookeeper-3.4.10]#bin/zkServer.shstatusJMXenabledbydefaultUsingconfig:/opt/module/zookeeper-3.4.10/bin/../conf/zoo.cfgMode:leader[root@hadoop104zookeeper-3.4.5]#bin/zkServer.shstatusJMXenabledbydefaultUsingconfig:/opt/module/zookeeper-3.4.10/bin/../conf/zoo.cfgMode:follower客戶端命令行操作命令基本語法功能描述help顯示所有操作命令lspath[watch]使用ls命令來查看當(dāng)前znode中所包含的內(nèi)容ls2path[watch]查看當(dāng)前節(jié)點(diǎn)數(shù)據(jù)并能看到更新次數(shù)等數(shù)據(jù)create普通創(chuàng)建-s含有序列-e臨時(shí)（重啟或者超時(shí)消失）getpath[watch]獲得節(jié)點(diǎn)的值set設(shè)置節(jié)點(diǎn)的具體值stat查看節(jié)點(diǎn)狀態(tài)delete刪除節(jié)點(diǎn)rmr遞歸刪除節(jié)點(diǎn)1）啟動(dòng)客戶端[atguigu@hadoop103zookeeper-3.4.10]$bin/zkCli.sh2）顯示所有操作命令[zk:localhost:2181(CONNECTED)1]help查看當(dāng)前znode中所包含的內(nèi)容[zk:localhost:2181(CONNECTED)0]ls/[zookeeper][zk:localhost:2181(CONNECTED)1]ls2/[zookeeper]cZxid=0x0ctime=ThuJan0108:00:00CST1970mZxid=0x0mtime=ThuJan0108:00:00CST1970pZxid=0x0cversion=-1dataVersion=0aclVersion=0ephemeralOwner=0x0dataLength=0numChildren=15）創(chuàng)建普通節(jié)點(diǎn)[zk:localhost:2181(CONNECTED)2]create/app1"helloapp1"Created/app1[zk:localhost:2181(CONNECTED)4]create/app1/server101"01"Created/app1/server101[zk:localhost:2181(CONNECTED)6]get/app1helloapp1cZxid=0x20000000actime=MonJul1716:08:35CST2017mZxid=0x20000000amtime=MonJul1716:08:35CST2017pZxid=0x20000000bcversion=1dataVersion=0aclVersion=0ephemeralOwner=0x0dataLength=10numChildren=1[zk:localhost:2181(CONNECTED)8]get/app1/server10101cZxid=0x20000000bctime=MonJul1716:11:04CST2017mZxid=0x20000000bmtime=MonJul1716:11:04CST2017pZxid=0x20000000bcversion=0dataVersion=0aclVersion=0ephemeralOwner=0x0dataLength=13numChildren=07）創(chuàng)建短暫節(jié)點(diǎn)[zk:localhost:2181(CONNECTED)9]create-e/app-emphemeral8888[zk:localhost:2181(CONNECTED)10]ls/[app1,app-emphemeral,zookeeper][zk:localhost:2181(CONNECTED)12]quit[atguigu@hadoop104zookeeper-3.4.10]$bin/zkCli.sh[zk:localhost:2181(CONNECTED)0]ls/[app1,zookeeper]app2[zk:localhost:2181(CONNECTED)11]create/app2"app2"[zk:localhost:2181(CONNECTED)13]create-s/app2/aa888Created/app2/aa0000000000[zk:localhost:2181(CONNECTED)14]create-s888Created/app2/bb0000000001[zk:localhost:2181(CONNECTED)15]create-s/app2/cc888Created/app2/cc000000000211[zk:localhost:2181(CONNECTED)16]create-s/app1/aa888Created/app1/aa0000000001[zk:localhost:2181(CONNECTED)2]set/app199910）節(jié)點(diǎn)的值變化監(jiān)聽104節(jié)點(diǎn)數(shù)據(jù)變化[zk:localhost:2181(CONNECTED)26]get/app1watch103/app1節(jié)點(diǎn)的數(shù)據(jù)[zk:localhost:2181(CONNECTED)5]set/app1 777104WATCHER::WatchedEventstate:SyncConnectedtype:NodeDataChangedpath:/app111）節(jié)點(diǎn)的子節(jié)點(diǎn)變化監(jiān)聽（路徑變化）104節(jié)點(diǎn)的子節(jié)點(diǎn)變化[zk:localhost:2181(CONNECTED)1]ls/app1watch[aa0000000001,server101]103主機(jī)/app1[zk:localhost:2181(CONNECTED)6]create/app1/bb666Created/app1/bb104WATCHER::WatchedEventstate:SyncConnectedtype:NodeChildrenChangedpath:/app112）刪除節(jié)點(diǎn)[zk:localhost:2181(CONNECTED)4]delete/app1/bb13）遞歸[zk:localhost:2181(CONNECTED)7]/app214）[zk:localhost:2181(CONNECTED)12]stat/app1cZxid=0x20000000actime=MonJul1716:08:35CST2017mZxid=0x200000018mtime=MonJul1716:54:38CST2017pZxid=0x20000001ccversion=4dataVersion=2aclVersion=0ephemeralOwner=0x0dataLength=3numChildren=2API應(yīng)用eclipse環(huán)境搭建創(chuàng)建一個(gè)Maven工程<dependencies><dependency><groupId>junit</groupId><dependencies><dependency><groupId>junit</groupId><artifactId>junit</artifactId><version>RELEASE</version></dependency><dependency><groupId>org.apache.logging.log4j</groupId><artifactId>log4j-core</artifactId><version>2.8.2</version></dependency><!--/artifact/org.apache.zookeeper/zookeeper--><dependency><groupId>org.apache.zookeeper</groupId><artifactId>zookeeper</artifactId><version>3.4.10</version></dependency></dependencies>perties文件到項(xiàng)目根目錄perties創(chuàng)建ZooKeeper客戶端privatestaticStringprivatestaticStringconnectString="hadoop102:2181,hadoop103:2181,hadoop104:2181";privatestaticintprivatestaticintsessionTimeout=2000;privateZooKeeperzkClient=null;@Beforepublicvoidinit()throwsException{zkClient=newZooKeeper(connectString,sessionTimeout,newWatcher(){@Overridepublicvoidprocess(WatchedEventevent){//收到事件通知后的回調(diào)函數(shù)（用戶的業(yè)務(wù)邏輯）System.out.println(event.getType()+"--"+event.getPath());//再次啟動(dòng)監(jiān)聽try{zkClient.getChildren("/",true);}catch(Exceptione){e.printStackTrace();}}});}//創(chuàng)建子節(jié)點(diǎn)@Testpublicvoidcreate()throwsException{////創(chuàng)建子節(jié)點(diǎn)@Testpublicvoidcreate()throwsException{////14String nodeCreated = zkClient.create("/eclipse", "hello Ids.OPEN_ACL_UNSAFE,CreateMode.PERSISTENT);}獲取子節(jié)點(diǎn)并監(jiān)聽////獲取子節(jié)點(diǎn)@TestpublicvoidgetChildren()throwsException{List<String>children=zkClient.getChildren("/",true);for(Stringchild:children){System.out.println(child);}//延時(shí)阻塞Thread.Thread.sleep(Long.MAX_VALUE);}//znodepublicvoidexist()throwsException{Statstat=zkClient.exists("/eclipse",false);System.//znodepublicvoidexist()throwsException{Statstat=zkClient.exists("/eclipse",false);System.out.println(stat==null?"notexist":"exist");}案例實(shí)戰(zhàn)監(jiān)聽服務(wù)器節(jié)點(diǎn)動(dòng)態(tài)上下線案例需求分析服務(wù)器動(dòng)態(tài)上下線案例分析業(yè)務(wù)功能服務(wù)器1服務(wù)器2服務(wù)器3業(yè)務(wù)功能服務(wù)器1服務(wù)器2服務(wù)器331服務(wù)端啟動(dòng)時(shí)去注冊信息（創(chuàng)建都是臨時(shí)節(jié)點(diǎn)）業(yè)務(wù)功能業(yè)務(wù)功能Zookeeper集群客戶端2客戶端32getChildren客戶端2客戶端3

客戶端14服務(wù)器節(jié)點(diǎn)上下線事件通知

/servers節(jié)點(diǎn)[zk:localhost:2181(CONNECTED)10]create/servers"servers"Created/serverspackagecom.atguigu.zkcase;packagecom.atguigu.zkcase;importjava.io.IOException;importorg.apache.zookeeper.CreateMode;importorg.apache.zookeeper.WatchedEvent;importorg.apache.zookeeper.Watcher;importorg.apache.zookeeper.ZooKeeper;importorg.apache.zookeeper.ZooDefs.Ids;publicclassDistributeServer{private static String connectString "hadoop102:2181,hadoop103:2181,hadoop104:2181";privatestaticintsessionTimeout=2000;privateZooKeeperzk=null;privateStringparentNode="/servers";//zkpublicvoidgetConnect()throwsIOException{zk=newZooKeeper(connectString,sessionTimeout,newWatcher(){@Overridepublicvoidprocess(WatchedEventevent){}});}//注冊服務(wù)器publicvoidregistServer(Stringhostname)throwsException{String create = zk.create(parentNode + "/server", Ids.OPEN_ACL_UNSAFE,CreateMode.EPHEMERAL_SEQUENTIAL);System.out.println(hostname+"isnoline"+create);}//業(yè)務(wù)功能publicvoidbusiness(Stringhostname)throwsException{System.out.println(hostname+"isworking...");Thread.sleep(Long.MAX_VALUE);}publicstaticvoidmain(String[]args)throwsException{//zk連接DistributeServerserver=newDistributeServer();server.getConnect();server.getConnect();//zk連接注冊服務(wù)器信息server.registServer(args[0]);//啟動(dòng)業(yè)務(wù)功能server.business(args[0]);}}"hadoop102:2181,hadoop103:2181,hadoop104:2181";privatestaticintsessionTimeout=2000;privateZooKeeperzk=null;"hadoop102:2181,hadoop103:2181,hadoop104:2181";privatestaticintsessionTimeout=2000;privateZooKeeperzk=nu