CEPH分布式存儲(chǔ)部署

CEPH分布式存儲(chǔ)部署PS：本文的所有操作均在mon節(jié)點(diǎn)的主機(jī)進(jìn)行，如有變動(dòng)另有注釋作者：網(wǎng)絡(luò)技術(shù)部徐志權(quán)口期：2014年2月10口VERSION1.0更新歷史：2011.2.10:首次完成gph部署文檔，塊設(shè)備及對(duì)象存儲(chǔ)的配置隨后添加。部署前網(wǎng)絡(luò)規(guī)劃1.1環(huán)境部署主機(jī)名公網(wǎng)IP(ethO)私網(wǎng)IP(ethl)操作系統(tǒng)運(yùn)行服務(wù)nodel0101CentOS6.5nion、mdsnode20202CentOS6.5osdnode30303CentOS6.5osd?操作系統(tǒng)使用CentOSG.5,因?yàn)橄到y(tǒng)己經(jīng)包含xfs的支持可以直接使用不需要再次編譯。由于CentOSG.5系統(tǒng)的內(nèi)核為2.6.32,因此要關(guān)閉碩盤的寫入緩存，若高于此版本不需要關(guān)閉。#hdparm~W0/dev/sdb0本次部署一共有一個(gè)監(jiān)控節(jié)點(diǎn)、一個(gè)元數(shù)據(jù)節(jié)點(diǎn)、兩個(gè)數(shù)據(jù)節(jié)點(diǎn)，每個(gè)數(shù)據(jù)節(jié)點(diǎn)擁有兩個(gè)硬盤作為數(shù)據(jù)盤。1.2網(wǎng)絡(luò)拓?fù)涔W(wǎng)主機(jī)名；nodol主機(jī)名；nodc2主機(jī)名；公網(wǎng)主機(jī)名；nodol主機(jī)名；nodc2主機(jī)名；nodc3私網(wǎng)1.3配置服務(wù)器、安裝ceph添加ceph的rpm庫key#rpm—importhttps：//ceph?com/git/?p二ceph?git;a=blob__plain;f二keys/release?asc'#rpm—importhttps：//ceph?com/git/?p二ceph?git;a=blobplain;f二keys/sutobuild?asc'?添加ceph-extras庫#vi/etc/yum.repos?d/ceph一extras_ceph-extras]name=CephExtrasPackagesbaseurl=http://ceph?com/packages/ceph一己xtras/rpm/centos6/$basearchenabled=lpriority=2gpgcheck=ltype=rpm"mdgpgkey=https://cEph?com/git/?p=ceph.git;a=blob_plain;f=keys/release?asc_ceph~extras~noarch]name=CephExtrasnoarchbaseurl=http://ceph?com/packages/ceph一己xtras/rpm/centos6/noarchenabled=lpriority=2gpgcheck=ltype=rpm"mdgpgkey=https://cEph?com/git/?p=ceph.git;a=blob_plain;f=keys/release?asc_ceph-ext:ras-source]name=CephExtrasSourcesbaseurl=http://ceph?com/packages/ceph一己xtras/rpm/centos6/SRPMSenabled=lpriority=2gpgcheck=ltype=rpm"mdgpgkey=https://cEph?com/git/?p=ceph.git;a=blob_plain;f=keys/release?asc添加ceph庫#rpm~Uvhhttp：//ceph?com/rpms/el6/noarch/ceph~release~l一0.el6?noarch?rpm添加epel庫#rpm~Uvhhttp://d1.fedoraproject?org/pub/epel/6/x86_64/epel一release一6一8?noarch?rpm安裝ceph#yumupdat己~y&&yuminstallceph~y

二*部署ceph2.1創(chuàng)建一個(gè)集群創(chuàng)建一個(gè)集群，集群名為ceph,在mon節(jié)點(diǎn)，這里是主機(jī)nodeb執(zhí)行以下操作2.1.1獲取一個(gè)uuid#uuidg^n2.1.2編輯配置文件#vi/etc/ceph/ceph.conf[global]auth_service_required=cephx#fi1estore_xattr_use__omap=true#該選項(xiàng)只針對(duì)extd文件系統(tǒng)格式auth_client_required=cephxauth_cluster_required=cephxauthsupported=nonefsid=a500528d~fc02~464e~b944-bddacbb5c3dl#填入剛才獲取到的uuidpublicnetwork=/24clusternetwork=/24osdpooldefauItsize=2osdpooldefauItminsize=1maxopenfiles=65535monosdfull:ratio=?95monosdnearfullratio=?85[mon]mondata=/datb/$cluste:r-$id[osd]osdjournalsize=1024osdjournal=/data/$name/journalosddata=/datE/$nam己2.2創(chuàng)建一個(gè)mon2.2.1編輯配置文件，添加mon參數(shù)#vi/etc/ceph/ceph.conf[mon?nodel]host=nodelmonaddr=01:67892.2.2創(chuàng)建一個(gè)monkeyring#ceph~authtool/etc/ceph/ceph.mon.keyring一一create一keyring一一gen~key一nmon.2.2.3創(chuàng)建mon數(shù)據(jù)目錄#mkdir-p/data/ceph-nodel#ceph~mon一inodel一一mkfs一一keyring/etc/ceph/ceph.mon.keyring2.2?4啟動(dòng)并查看集群狀態(tài)#/etc/init?d/cephstartmon.nodel#ceph-s出現(xiàn)以下提示，啟動(dòng)成功：[root?nodel、]#ceph-sclustera500528d~fc02~464e-b944"bddacbb5c3dlhealthHEALTH_ERR192pgsstuckinactive;192pgsstuckunclean;noosdsmonmapel:1monsat{nodel=192?168.100.101:6789/0},electionepoch2,quorum0nodelosdmapel:0osds:0up,0inpgmapv2:192pgs,3pools,0bytesdata,0objects0kBused,0kB/0kBavail192creating2.3創(chuàng)建一個(gè)osd首先為節(jié)點(diǎn)添加一塊硬盤（這里是主機(jī)node2）PS：硬件支持熱插拔的情況卜可以使用以下命令，在線檢測(cè)新増硬盤#echo”——M>/sys/class/scsijiost/host2/scaii2.3.1獲取osd的ED##cephosdcreate〃記錄得到的編號(hào)，該編號(hào)是下面創(chuàng)建的osd的ID2?3?2編輯配置文件#vi/etc/ceplvceph.conf添加[osd.O]publicaddi=02clusteraddi=022.3.3同步配置文檔到節(jié)點(diǎn)node2#scp-rroot?01:/etc/ceplvroot?02:/etc/2.3.4部署osd節(jié)點(diǎn)#sshioot@192?168.100.102^parted/dev/sdbniktablegpt^parted/dev/sdbnikpartosd.O120g〃新加的碩盤為20g,并將所有空河劃分為一個(gè)分區(qū)#nikfs-txfs/dev/sdb1#nikdk/data/osd.OAmount/dev/sdb1/data/osd.O#ceph-osd-i0-nikfs-nikkey#cephauthaddosd.Oosd'allow**mon'allowrwx'-i/data/osd.O/keyiiiig#cephosdcmshaddosd.O0.2root=defaulthost=node22.3.5啟動(dòng)osd進(jìn)程#ceph-osd-i023.6查看進(jìn)程#ps-ef|grepceph-osd以下提示啟動(dòng)成功[root@node2?]#ps-efjgrepceph-osdroot13641008:42?00:00:07ceph-osd-102.3.7查看osd狀態(tài)#exit#cephosdstatosd添加成功[root@nodel?]#cephosdstatosdinape41:1osds:1up,1in2.3.8開機(jī)自動(dòng)啟動(dòng)#vi/etc/fstab添加/dev/sdb1/data/osd.Oxfsdefaults00#echoMceph-osd-i0"?/etc/rc.local2.4創(chuàng)建一個(gè)mds創(chuàng)建mds#ceph-mds-inodel-nmds.node1?c/etc/ceplvceph.conf-m192?168.100.101:6789檢查mds狀態(tài)，成功啟動(dòng)如下[root@nodel?]#cephmdsstate!9:1/1/1up{0=node1=up:active}三、配置ceph客戶端3.1確定操作系統(tǒng)內(nèi)核3.1.1查看內(nèi)核版本#uname-1若內(nèi)核版本低于2.6.32時(shí)，建議升級(jí)到2.6.34或以上3丄2更新內(nèi)核，編譯ceph模塊這里以升級(jí)到3.10.29為例#wg筑https://www.keni亡l?OT0pi】blinux/keniel/v3?x/1idi】x-3?10?29?tar.xz#tar-Jxvflinux-3.10.29.tar.xz#cdliiiux-3.10.29#cp/boot/coiifig-2.6.32-431.3.1.el6.x86_64./.config〃將當(dāng)前系統(tǒng)內(nèi)核的配置文件復(fù)制到kemel源文件的目錄內(nèi)彳makemenuconfig依次選擇下面選項(xiàng)，ceph必須編譯為模塊MFilesystemsN亡twoikFileSystemsCephdistributedfilesystemRPC：Enabledprintkdebugging<M>cifssupportCadvanced^nework保存并退出彳make彳makemodules_iiistall彳makeinstall確定boot木有下存在以下文件[root@nodellinux-3.10.29]#11/boot/|grep3.10initranifs-3.10.29.imgSystem.map->/boot/Svstem.map-3.10.29System.map-3.10.29vnilinuz->/boot/vinlinuz-3.10.29vnilinuz-3.10.29更改gmb.conf#vi/etc/gmb.conf添加titleCentOS(3.10.29)root(hd0,0)kernel/vnilinuz-3.10.29ioroot=/dev/iiiapperA^blGroup-lv_.rootid_NO_LUKSLANG=enUS.UTF-8rdNOMDrdLVMLV=VolGioup/lvswapSYSFONT=lataicyrheb-sunl6ciaslikernel=autoid_LVM_LV=WlGioiip/lv_iootKEYBOARDTYPE=pcKEYTABLE=usid_NO_DMrhgbquietinitid/iiiitramfs-3.10.29.ung34配置文件目錄共享客戶端PS:確定已經(jīng)運(yùn)行一個(gè)ceph集群，并且集群包含至少一個(gè)或以上mds節(jié)點(diǎn)掛載ceph分布式存儲(chǔ)文件目錄共享以下在客戶端上操作#yuminstallceph-fiise-y#ceph-fiise-m01:6789/milt測(cè)試寫入1G文件#ddif=/dev/zeioof^/iiiiit/test.lgbs=8192count=125000U!I>cephU!I>ceph擴(kuò)容4.1增加osd節(jié)點(diǎn)（以下操作在nodel執(zhí)行?即mon節(jié)點(diǎn)）根據(jù)第一章在新的節(jié)點(diǎn)node3服務(wù)器安裝ceph,并配置好環(huán)境4.1.1獲取osd的ED##cephosdcreate//記錄得到的編號(hào)，該編號(hào)是下面創(chuàng)建的osd的ED4.1.2編輯配置文件#vi/etc/ceplvceph.conf添加[osd.l]publicaddr=03clusteraddr=034.1.3同步配置文檔到節(jié)點(diǎn)node3#scp-rroot@01:/etc/ceplvroot@03:/etc/4.1.4部署osd節(jié)點(diǎn)#sshroot@03^parted/dev/sdbniktablegpt^parted/dev/sdbnikpartosd.l120g〃新加的碩盤為20g,并將所有空間劃分為一個(gè)分區(qū)#nikfs-txfs/dev/sdb1#nikdii--p/data/osd.lAmount/dev/sdb1/data/osd.1#ceph-osd-i1--nikfs—nikkey#cephauthaddosd.losd'allow**monallowrwx°-i/data/osd.1/keyiiiig#cephosdcmshaddosd.l0.2root=defaulthost=node24.1.5啟動(dòng)osd進(jìn)程#ceph-osd-i14.1.6查看進(jìn)程#ps-ef|giepceph-osd以下提示啟動(dòng)成功[root@node3~]#ps-ef]grepceph-osdroot13641008:42?00:00:07ceph-osd-104.1.7查看osd狀態(tài)#exit#gphosdstatosd添加成功[root@nodel?]#cephosdstatosdmape41:2osds:2up,2in4.2增加mom以下操作在主機(jī)node2執(zhí)行）PS：每臺(tái)主機(jī)只能為wph集群?jiǎn)?dòng)一個(gè)moil進(jìn)程PS2：集群中只有兩個(gè)mon節(jié)點(diǎn)時(shí)，其中一個(gè)斷開會(huì)導(dǎo)致集群不能訪問，建議mon總數(shù)3個(gè)或以上。本次mon將部署在主機(jī)node24.2.3創(chuàng)建新的monitor數(shù)據(jù)目錄彳mkdir/data/ceph-node24.2.4部署新的mon#cephauthgetmon.-o/data/ceph-node2/ceph.mon.kevrmg#cephmongetmap-o/data-zceph-node2/ceph.mon.niap#ceph-mon-inode2-nikfs—moimiap/data//ceph-node2/ceph.mon.map-kevrmg\/data/ceph-node2/ceph.mon.keyrmg#cephmonaddnode202:67894.2.1啟動(dòng)新的mon#ceph-mon-inode2-public-addr02:6789五、檢查ceph集群狀態(tài)#cephhealth//ceph健康狀態(tài)#ciphstatus//ceph當(dāng)前全部狀態(tài)#ceph-w〃實(shí)時(shí)監(jiān)控ceph狀態(tài)及變化#ciphosddump〃所有osd詳細(xì)狀態(tài)#gphosdtree//osd所在位置，及狀態(tài)#ciphquoiiim_status//mon優(yōu)先級(jí)狀態(tài)#ciphmondump//moil節(jié)點(diǎn)狀態(tài)#ciphmdsdump//mds詳細(xì)狀態(tài)六、Openstack與Ceph整合6.1整合openstack與ceph6.L1創(chuàng)建池#cephosdpoolcreatevolumes128#cephosdpoolcreateimages128#cephosdpoolcreatebackups1286.1.2獲取ceph配置文件PS:應(yīng)用J?openstack所有節(jié)點(diǎn)#scp-r3:/etc/ceph/etc6.13所有openstack節(jié)點(diǎn)安裝ceph庫文件#yuminstall-vpython-cephceph6.1.4創(chuàng)建ceph用戶#cephauthget-or-createclient.cmdermon'allowFosd*allowclass-readobject_prefixrbd_childien.allowrwxpool=volumes,allowrxpool=miages*#cephauthget-or-createclient.glancemon'allowFosd'allowclass-readobject_prefixrbd_childien.allowrwxpool=iniages*#cephauthget-or-createclient.cmder-backupmon'allowFosd'allowclass-readobject_prefixrbd_childien.allowrwxpool=backups*6.1.5獲取ceph用戶的key加入到ceph配置文件中PS:應(yīng)用J?openstack所有節(jié)點(diǎn)#cephauthget-or-createclient.glance|tee/etc/ceplvceph.client.glaiice.keyringchownglance:glaiice/etc/ceplVceph.client.glance.keyiing#cephauthget-or-createclient.cmder|tee/etc/ceplVceph.client.cinder.keyiingchownciiide匚cind亡［/etc/ceplvceph.client.cmder.kevrmg#cephauthget-or-createclient.cmder-backup|tee/etc/ceph/ceph.client.ciiider-backup.keyrmgchownciiide匚cind亡［/etc/ceplvceph.client.cmdei-backup.keyriiig6.1.6配置計(jì)算節(jié)點(diǎn)的nova-compute使用cephPS:應(yīng),l|］T所有計(jì)篦節(jié)點(diǎn)#cephauthget-keyclient.cinderteeclient.ciiider.key#uuidgen3a37c634-3b97-698c-c0b0-cb6f01fa4533#cat>seciet.xiiil?EOF<secretephemeral=,no,private=,no,><uuid>3a37c634-3b97-698c-c0b0-cb6f01fa4533</uuid><usagetype=,ceph*><name>client.ciiidersecret</name></usage></seciet>EOF#viishsecret-define-filesecret.xmlSecret3a37c634-3b97-698c-c0b0-cb6f01fa4533created將上述所得uuid值保存至/etc/nova/nova.confibd_secret_uuid=3a37c634-3b97-698c-c0b0-cb6f01fa4533#viishsecret-set-value-secret3a37c634-3b97-698c-c0b0-cb6f01fa4533-base64$(catclient.cinder.key)6.1.7glance>cinder、nova的ceph配置Glance：default_stoie=rbdibd_store_usei-glancerbd_store_pool=unagesshow_iinage_duect_url=TiiieCinder：volume_dnver=cmder.