使用BOSH進行CloudFoundry生產環(huán)境部署

1、用BOSH自動部署大規(guī)模Cloud FoundryHenry Zhang, VMWare, China張軒寧的圍脖Jacky Li, VMWare, China演講內容Cloud Foundry云平臺簡介Cloud Foundry架構Cloud Foundry部署模型BOSH工作原理用BOSH自動化部署Cloud Foundry用BOSH實現(xiàn)Cloud Foundry彈性BOSH應用實例Cloud Foundry 云平臺簡介Sound architecture: future proofing without overdoingUnit and integration testsRespond

2、ing to changing business needsConfusing technology landscape應用的復雜度: 開發(fā)4Choosing the right hardware, operating system, web serverMonitoring applicationsResponding to scalability needsDealing with hardware- and system-level failuresUpgrading without substantial down time應用的復雜度：部署和運維5內在的和實際的復雜度Implemen

3、tationFunctional logicImplementation overhead內在的復雜度實際的復雜度 怎么辦?6云計算的三個層次7Application Service InterfacePrivate Clouds PublicCloudsMicro CloudsCloud Foundry 開放 PaaSData ServicesOther ServicesMsg ServicesCloud Provider Interface8Cloud Provider InterfaceApplication Service InterfacePrivate Clouds PublicC

4、loudsData ServicesOther ServicesMsg ServicesPartnersApache2 licenseMicro CloudsCloud Foundry 開放 PaaS9Cloud Foundry 的架構Cloud Foundry 層次Hardware (CPU, Memory, Storage, Network)Infrastructure as a ServiceOuter Shell (BOSH)Inner Shell (CF)Clients11Cloud Foundry 內殼（Inner Shell）RoutersCloudControllersAppA

5、ppHealthManagerExecution Agents (DEA) PoolMessagingUsersDevelopersStagersServices12Cloud Foundry內部架構stagingjobsrediscc - dbcloud controllerstagerpackage cacheblobstoreuaa - dbdeahealthmanagerstaging logsuaa - AuthNApp meta datadeadeadeadeadea可動態(tài)發(fā)現(xiàn)組件組件之間無依賴性啟動順序無關各組件獨立的擴展設計原則14Cloud Foundry部署模型Cloud

6、Foundry 組件模型uaaccdbnatshealth manager16Cloud Foundry 部署模型（1）- 單結點單虛擬機或物理機OS17VMstagerCloud Foundry 部署模型（2）- 多結點VMuaaVMserviceVMcloud controllerVMccdbVMnatsVMrouterVMhealth managerVMrouterVMserviceVMserviceVMserviceVMcloud controllerVMdeaVMdeaVMdeaVMdeaVMstagerVMstager每臺虛擬機（或物理機）部署一個或數(shù)個組件18500 5,000

7、VMs40+ unique node types75+ unique software packages75+ unique environmentsSmall teamCloudF生產系統(tǒng)部署的實例dev_setup 手工部署直接在操作系統(tǒng)(ubuntu 10.04)中用腳本部署可部署單結點或多結點適合小規(guī)模實驗的環(huán)境支持虛擬機和物理機每臺機器需要手工安裝和修改配置文件系統(tǒng)擴容能力和彈性計算較差Cloud Foundry兩種部署方法（1）BOSH 自動部署自動化大規(guī)模部署多結點系統(tǒng)集中式的配置文件支持自動化系統(tǒng)擴容僅支持虛擬機對各類IaaS平臺集成 (CPI)在C生產系統(tǒng)中使用Cloud F

8、oundry兩種部署方法（2）BOSH工作原理BOSH is a recursive definition of BOSH Outter SHell什么是BOSH?來自Cloud Foundry的開源項目通用的自動化部署工具鏈可以部署多種軟件Cloud Foundry, Gerrit, Jenkins, Wordpress, MySQL, BOSH是大規(guī)模自動化管理工具部署系統(tǒng)更新配置版本管理監(jiān)控運行BOSH在Cloud Foundry中的作用BOSH的內部結構BOSH部署原理（1）BOSH CLIOperationStaffBOSH BOSH是部署系統(tǒng)的總指揮管理員通過BOSH CLI工具發(fā)

9、送指令給BOSHBOSH部署原理（2）BOSH CLIIaaSOperationStaff創(chuàng)建虛擬機BOSH CPICPI ( Cloud Provider Interface)支持多種 IaaSBOSH通過CPI管理IaaS層資源（虛擬機、虛擬存儲、虛擬網(wǎng)絡等）BOSH部署原理（3）Inner Shell. . .BOSH CLIVMOperationStaff創(chuàng)建虛擬機VMVMVMBOSH調用CPI動態(tài)生成虛擬機Inner Shell 被管理的部分IaaSBOSH CPIBOSH部署原理（4）Inner Shell. . .BOSH CLIVMStemcellOperationStaffD

10、eployment創(chuàng)建虛擬機VMVMVMStemcell提供虛擬機模板Deployment manifest定義部署細節(jié): IP地址、用戶名/密碼等IaaSBOSH CPIBOSH部署原理（5）Inner Shell. . .BOSH CLIVMAgentStemcellDeploymentOperationStaff創(chuàng)建虛擬機VMAgentVMAgentVMAgent每個虛擬機內嵌一個Agent負責管理、控制、監(jiān)控虛擬機IaaSBOSH CPIBOSH部署原理（6）Inner Shell. . .BOSH CLIVMAgentStemcellDeploymentOperationStaff創(chuàng)

11、建虛擬機VMAgentVMAgentVMAgentAgent接收BOSH的指令完成軟件部署安裝、應用起停、系統(tǒng)監(jiān)控等任務IaaSBOSH CPIBOSH部署原理（7）Inner Shell. . .BOSH CLIVMAgentStemcellReleaseDeploymentOperationStaff創(chuàng)建虛擬機VMAgentVMAgentVMAgentRelease包含軟件的源代碼、目標碼、版本、配置等信息IaaSBOSH CPICloud Foundry平臺的升級過程滾動式升級：無狀態(tài)(Stateless)組件Message BusStarting with v1 VMsRouterIn

12、coming HTTP RequestsMessage BusCloudControllerv2Deploy a v2 VM (canary)CreateRouterIncoming HTTP RequestsMessage BusCloudControllerv2If it works, add more v2 VMs.CreateRouterIncoming HTTP Requests例子：Cloud Controller結果:從v1升級到v2, 創(chuàng)建了新的虛擬機，中間沒有宕機時間。Message BusCloudControllerv1CloudControllerv2 while re

13、moving v1 VMsCreateDestroyRouter / LBIncoming HTTP RequestsMessage Bus until all VMs are v2Router / LBIncoming HTTP Requests并行升級金絲雀（Canaries）最終一致性（Eventual consistency）部署要點無狀態(tài)(stateless)VM沒有持久化數(shù)據(jù)，升級時可以直接銷毀（用新VM代替）有狀態(tài)(stateful)的VM具有持久化的磁盤（如數(shù)據(jù)庫數(shù)據(jù)）升級時需要保留持久化的磁盤有狀態(tài)和無狀態(tài)的VM代碼升級（不涉及數(shù)據(jù)）Stateful VM 升級VM 更新用B

14、OSH自動化部署Cloud Foundry(vSphere)BOSH的內部結構BOSH由6個虛擬機組成安裝BOSH的步驟安裝BOSH CLI （Ubuntu 10.04)部署Micro BOSH (單個VM)部署多結點分布式BOSH (6個VMs)BOSHVMVMVMVMVMVMMicro BOSH VMBOSHCLI準備工作 - 軟件64-bit Ubuntu 10.04 LTS, better in ISO format.vSphere V4.1 or V5.x, vSphere clientvCenter (installed on a Win2008 R2 64bit or a Win

15、2003 server, physical or virtual machine)準備工作 硬件（1）* Cloud Foundry需要最少的結點數(shù)，實際中可能會更多。準備工作 硬件（2）物理服務器配置估算：假定，每臺虛擬機4GB內存，1 vCPU, 則大約需要 4 x 43 = 172 GB RAM, 1 x 43 = 43 vCPU，因此可以選用 6 臺物理服務器：每臺 8 核CPU，32GB RAM.較小配置 （部署實驗環(huán)境）每臺虛擬機256MB內存，1臺物理服務器： 8 核CPU，16GB RAM.注：CPU在試驗環(huán)境中不是最關鍵資源準備工作 網(wǎng)絡建議分配4個VLAN足夠的IP地址結點

16、之間可以連通，沒有防火墻vSphere網(wǎng)絡名稱必須相同必須直接連接 Internet （很重要?。￢LANNodesManagement VLANHypervisors and NFS storageCF VLANBOSH VMs, and VMs of cloud foundryService VLANFor LB, dual-home routers Public VLANFor LB, incoming requests準備工作 網(wǎng)絡 (2) 參考網(wǎng)絡拓撲圖準備工作 網(wǎng)絡 (3) vSphere 配置Cluster準備工作 存儲建議用NFS存儲所有的vSphere在vCenter中組成一

17、個Cluster所有vSphere掛接同樣的NFS mount point在vSphere中存儲必須用同樣的名字vSpherevSpherevSpherevSphereNFS準備工作 存儲（2） vSphere 配置安裝 BOSH CLI創(chuàng)建一個干凈的虛擬機安裝Ubuntu 10.04安裝Ruby環(huán)境安裝BOSH CLI工具記住： 虛擬機要直接連接Internet安裝 Micro BOSH 通過BOSH CLI里面的工具Bosh_deployer，安裝Micro BOSH完成后, vCenter中可以看到Micro BOSH虛擬機記?。?虛擬機要直接連接InternetDEMO: Stemce

18、ll的下載$ bosh public stemcells$ bosh download public stemcell micro-bosh-stemcell-0.1.0.tgz安裝 BOSH 通過Micro BOSH安裝分布式 BOSH （6 VMs)需要規(guī)劃6臺虛擬機的網(wǎng)絡以及存儲規(guī)劃Resource PoolsPool NameSizeConfigurationJobssmall3RAM:512MB, CPU:1, DISK:2GBnats, redis, health_monitormedium2RAM:1GB, CPU: 1, DISK: 8GBpostgres, blobstore

19、director1RAM:2GB, CPU: 2, DISK: 8GBdirector安裝 BOSH （2） 每個job占用一臺虛擬機把job分配到Resource PoolJobResource_poolIPnatssmall20postgresmedium19redissmall18directordirector17blob_storemedium16health_monitorsmall15安裝 BOSH （3） 編寫deployment manifest的yml文件參考gitHub上面的樣板啟動部署安裝 Cloud Foundry通過BOSH安裝分布式 Cloud Foundry （

20、34+ VMs)需要規(guī)劃所有虛擬機的網(wǎng)絡以及存儲規(guī)劃Resource PoolsPool NameSizeConfigurationJobssmall32RAM:1GB, CPU: 1, DISK: 8GBNats， router,ccdb_postgresuaadbvcap_redisUaa等30個jobsmedium4RAM:2GB, CPU: 2, DISK: 16GBdebian_nfs_servercloud_controllerdea安裝 Cloud Foundry （2） 規(guī)劃每個job的配置：實例數(shù)、資源分配、網(wǎng)絡等可選的必要的. . . . . . 安裝 Cloud Foun

21、dry（3） 編寫deployment manifest參考gitHub上面的樣板啟動Cloud Foundry部署DEMO下載Cloud Foundry release開源代碼Create releaseUpload releaseDeploy CFBOSH部署完成之后的Cloud Foundry (vCenter)BOSH部署完成之后的Cloud Foundry (BOSH CLI)BOSH的監(jiān)控功能虛擬機里的agent發(fā)送心跳信息采集系統(tǒng)數(shù)據(jù)數(shù)據(jù)匯總64實現(xiàn)Cloud Foundry彈性計算Cloud Foundry 云平臺的彈性Open source Platform as a Ser

22、vice projectApp InstancesServices包含整個Cloud Foundry云平臺單虛擬機大規(guī)模部署Cloud Foundry云平臺多達數(shù)千個虛擬機結點BOSH動態(tài)增加DEA結點，可承載更多云應用BOSH動態(tài)增加Service結點，可提供更多云服務Cloud Foundry 云平臺彈性的原理IaaSDEAIaaSDEAIaaSSvcBOSHBOSH動態(tài)增加DEA結點，可承載更多云應用BOSH動態(tài)增加Service結點，可提供更多云服務Cloud Foundry 云平臺彈性的原理IaaSDEAIaaSDEAIaaSSvcDEADEASvcDEADEASvcDEADEASvcDEADEASvcBOSHBOSH實現(xiàn)Cloud Foundry平臺彈性 更新deployment manifest中dea和services