京東JDOS平臺(tái)與敏捷數(shù)據(jù)中心

1、京東JDOS2.0平臺(tái)與敏捷數(shù)據(jù)中心容器技術(shù)在京東商城歷史與發(fā)展JDOS2.0平臺(tái)京東敏捷數(shù)據(jù)中心建設(shè) 開(kāi)源計(jì)劃Q&AJDOS 1.03l分物理機(jī) = 分容器l業(yè)務(wù)系統(tǒng)100%全面容器運(yùn)行 l上線周期由1天降低到10分鐘 l少買3/4機(jī)器JD Data Center Operating System (JDOS)Physical serversVMsContainersJFSOVSSSJD Linux KernelJDOS 發(fā)展歷程42014201520162017JDOS1.0推出，推進(jìn)業(yè)務(wù)容器化推出JDOS平臺(tái)將業(yè)務(wù)逐漸接入容器業(yè)務(wù)間通過(guò)容器實(shí)現(xiàn)隔離業(yè)務(wù)容器化全面落地支撐雙11/618積累

2、了大量的容器運(yùn)營(yíng)經(jīng)驗(yàn)對(duì)Linux內(nèi)核進(jìn)行了深度定制 全面建設(shè)JDOS 1.0，實(shí)現(xiàn)了容器分配的秒級(jí)伸縮JDOS容器生態(tài)建設(shè)完善容器的監(jiān)控、網(wǎng)絡(luò)、存儲(chǔ)等容器生態(tài)建設(shè) 建設(shè)了鏡像中心，為平臺(tái)化鋪平道路積累了豐富的運(yùn)營(yíng)監(jiān)控?cái)?shù)據(jù)JDOS2.0全面推出從基礎(chǔ)設(shè)施轉(zhuǎn)向應(yīng)用平臺(tái)推進(jìn)了業(yè)務(wù)從源碼到編譯打包到鏡像的全流程逐步解除對(duì)部署系統(tǒng)的依賴 建設(shè)了平臺(tái)化的JDOS 2.0，實(shí)現(xiàn)業(yè)務(wù)層級(jí)的秒級(jí)伸縮容器技術(shù)在京東商城歷史與發(fā)展JDOS2.0平臺(tái)京東敏捷數(shù)據(jù)中心建設(shè) 開(kāi)源計(jì)劃Q&AJDOS 2.0集群部署架構(gòu)自動(dòng)構(gòu)建NodeDockerfileSource CodeJenkins MasterJenkins Sl

3、ave PodAppAppAppSkyFSSkyNetElasticSearchPrometheusJDOS-apiSkyDNSapicontrollerscheduler鏡像中心 Docker ImageApp集群管理kubeletNetwork Volume Log MonitorApp生產(chǎn)環(huán)境測(cè)試環(huán)境AppSkyLB6JDOS 2.0典型業(yè)務(wù)7SkyDNSDNS ServerAPICheck IP statusetcdService to DNSWrite to/DeletedockerRead DNS fromDNS req/respIP port scanRead DNS from

4、/Write DNS to /Del/UpdateRed IP from /Update IP statusUserGet ServiceJDOS2.08Load BalancerIP port scanSkyLBLoad Balancer.DDocokcekrerag entag ent ag entDockerKubernetesLoad BalancerDDocokcekrerag entag ent ag entDockerLoad BalancerDDocokcekrerag entag entag entDocker.Load Balancer Based on DPDKDDoco

5、kcekrerag entag ent ag entDPDKLoad BalancerLoad BalancerDDocokcekrerag entag ent ag entDPDKLoad BalancerDDocokcekrerag entag entag entDPDKKubernetes masterLoad Balance Control CentergRPCgRPC Watch ServiceCreate/Del/Get RC for Service9SkyLBRouterSkylbSkylbquaggaDPDKKNILoad balancerSkylbquaggaDPDKKNIL

6、oad balancerService EndpointService EndpointService EndpointBGPBGPBGPVIP:545454Router Table:Dest: viia: Dest: viia: Dest: viia: Dest: viia: Dest: viia: Dest: viia: Dest: viia: Dest: viia:10.20.4.2Dest:10.20.5.0 viia:10.20.5.2quaggaDPDKKNILoad balancer10ContainerFSVFSFUSELibFuseRaftBlockID-xDISKDISKB

7、lockID-xBlockID-xBlockID-xBlockID-xBlockID-xBlockGroupBlockID-xBlockID-xMataData Table ( raft group )VolumeVolumemanagerMata nodesData nodesclientappmetanode Leadermetanode Fllowermetanode FllowerKeparent inoy :+ filenameValueInode attr:ibutesDatanode1Datanode3BlockID-xDISKBlockID-xBlockID-xBlockID-

8、xDatanode2BlockGroupBlockGroupBlockGroup11容器技術(shù)在京東商城歷史與發(fā)展JDOS2.0平臺(tái)京東敏捷數(shù)據(jù)中心建設(shè) 開(kāi)源計(jì)劃Q&AJDOS 敏捷數(shù)據(jù)中心是什么a PaaS platforma cluster managera container platforman operating system13JDOS 敏捷數(shù)據(jù)中心14JDOS 是中間件服務(wù)平臺(tái)JDOSJFS獨(dú)享集群ESJIMDBSimple API Share powerful ClusterJMQJSSJED15JDOS是組件管理平臺(tái)Universe CatalogRepopackage用戶A的

9、zookeeper集群JDOS 廊坊一區(qū)集群用戶A的kafka broker集群Kafka chart模板管理變量管理依賴管理Kafka Chart服務(wù)說(shuō)明容器的鏡像、啟動(dòng)命令，負(fù)載均衡配置等Kafka broker的數(shù)量等依賴zookeeper服務(wù)Kafka服務(wù)的介紹ZookeeperKafka brokerZookeeper chart容器的鏡像、啟動(dòng)命令，負(fù)載均衡配置等Zookeeper的數(shù)量等無(wú)依賴服務(wù)Zookeeper服務(wù)的介紹模板管理變量管理依賴管理Zookeeper Chart服務(wù)說(shuō)明ZookeeperZookeeperKafka brokerKafka brokerKafka brokerKafka brokerKafka brokerJDOS 黃村一區(qū)集群16用戶A的zookeeper集群用戶A的kafka br