mdadm規(guī)范性操作流程

基本語法： mdadm mode options 目前支持： LINEAR, RAID0(striping), RAID1(mirroring), RAID4, RAID5, RAID6, RAID10, MULTIPATH和FAULTY模式(7種)：Assemble：加入一個以前定義的陣列Build：創(chuàng)建一個沒有超級塊的陣列Create：創(chuàng)建一個新的陣列，每個設備具有超級塊Manage： 管理陣列(如添加和刪除)Misc：允許單獨對陣列中的某個設備進行操作(如停止陣列)Follow or Monitor:監(jiān)控RAID的狀態(tài)Grow：改變RAID的容量或陣列中的設備數(shù)目選項：-A, -assemble：加入一個以前定義的陣列-B, -build：創(chuàng)建一個沒有超級塊的陣列(Build a legacy array without superblocks.)-C, -create：創(chuàng)建一個新的陣列-F, -follow, -monitor：選擇監(jiān)控(Monitor)模式-G, -grow：改變激活陣列的大小或形態(tài)-I, -incremental：添加一個單獨的設備到合適的陣列，并可能啟動陣列-auto-detect：請求內(nèi)核啟動任何自動檢測到的陣列-h, -help：幫助信息，用在以上選項后，則顯示該選項信息-help-options：顯示更詳細的幫助-V, -version：打印mdadm的版本信息-v, -verbose：顯示細節(jié)-b, -brief：較少的細節(jié)。用于 -detail 和 -examine 選項-Q, -query：查看一個device，判斷它為一個 md device 或是 一個 md 陣列的一部分-D, -detail：打印一個或多個 md device 的詳細信息-E, -examine：打印 device 上的 md superblock 的內(nèi)容-c, -config= ：指定配置文件，缺省為 /etc/mdadm.conf-s, -scan：掃描配置文件或 /proc/mdstat以搜尋丟失的信息。配置文件/etc/mdadm.conf使用mdadm創(chuàng)建RAID5Create (mdadm -create)模式用來創(chuàng)建一個新的陣列。 在這里我們首先使用mdadm -create -help查看一下幫助：rootlocalhost mdadm-2.6.2# mdadm -create -helpUsage: mdadm -create device -chunk=X -level=Y -raid-devices=Z devicesThis usage will initialise a new md array, associate somedevices with it, and activate the array. In order to create anarray with some devices missing, use the special word missing inplace of the relevant device name.Before devices are added, they are checked to see if they already containraid superblocks or filesystems. They are also checked to see ifthe variance in device size exceeds 1%.If any discrepancy is found, the user will be prompted for confirmationbefore the array is created. The presence of a -run can override thiscaution.If the -size option is given then only that many kilobytes of eachdevice is used, no matter how big each device is.If no -size is given, the apparent size of the smallest drive givenis used for raid level 1 and greater, and the full device is used forother levels.Options that are valid with -create (-C) are: -bitmap= : Create a bitmap for the array with the given filename -chunk= -c : chunk size of kibibytes -rounding= : rounding factor for linear array (=chunk size) -level= -l : raid level: 0,1,4,5,6,linear,multipath and synonyms -parity= -p : raid5/6 parity algorithm: left,right-,asymmetric -layout= : same as -parity -raid-devices= -n : number of active devices in array -spare-devices= -x: number of spares (eXtras) devices in initial array -size= -z : Size (in K) of each drive in RAID1/4/5/6/10 - optional -force -f : Honour devices as listed on command line. Dont : insert a missing drive for RAID5. -run -R : insist of running the array even if not all : devices are present or some look odd. -readonly -o : start the array readonly - not supported yet. -name= -N : Textual name for array - max 32 characters -bitmap-chunk= : bitmap chunksize in Kilobytes. -delay= -d : bitmap update delay in seconds.接下來我們使用mdadm創(chuàng)建在/dev/md0上創(chuàng)建一個由sdb、sdc、sdd3塊盤組成(另外1塊盤sde為熱備)的RAID5：rootlocalhost mdadm-2.6.2# mdadm -create -verbose /dev/md0 -level=raid5 -raid-devices=3 /dev/sdb /dev/sdc /dev/sdd -spare-devices=1 /dev/sdemdadm: layout defaults to left-symmetricmdadm: chunk size defaults to 64Kmdadm: size set to 8388544Kmdadm: array /dev/md0 started.每個mdadm的選項都有一個縮寫的形式，例如，上面我們創(chuàng)建RAID 5的命令可以使用下列的縮寫形式：rootlocalhost mdadm-2.6.2# mdadm -Cv /dev/md0 -l5 -n3 /dev/sdb /dev/sdc /dev/sdd -x1 /dev/sde二者的效果是相同的。切記：此處需建立配置文件，如果沒有配置文件，在停止raid后就無法再激活rootmylab # echo DEVICE /dev/sdb5-8 /etc/mdadm.confrootmylab # mdadm -Ds /etc/mdadm.conf 沒有配置文件的測試：rootmylab # mv /etc/mdadm.conf /etc/mdadm.conf.oldrootmylab # mdadm -S /dev/md0 mdadm: stopped /dev/md0rootmylab # mdadm -A /dev/md0 mdadm: /dev/md0 not identified in config file.rootmylab # mv /etc/mdadm.conf.old /etc/mdadm.confrootmylab # mdadm -A /dev/md0 mdadm: /dev/md0 has been started with 3 drives and 1 spare.查看RAID狀態(tài)接下來我們使用cat /proc/mdstat命令來查看一下RAID的狀態(tài)，我們也可以利用watch命令來每隔一段時間刷新/proc/mdstat的輸出。使用CTRL+C可以取消。rootlocalhost mdadm-2.6.2# watch -n 10 cat /proc/mdstatEvery 10s: cat /proc/mdstat Thu May 24 11:53:46 2007Personalities : raid5read_ahead 1024 sectorsmd0 : active raid5 sdd4 sde3 sdc1 sdb0 16777088 blocks level 5, 64k chunk, algorithm 2 3/2 UU_ =. recovery = 24.0% (2016364/8388544) finish=10.2minspeed=10324K/secunused devices: rootlocalhost mdadm-2.6.2#接下來我們?yōu)殛嚵袆?chuàng)建文件系統(tǒng)：rootlocalhost mdadm-2.6.2# mkfs.ext3 /dev/md0mke2fs 1.34 (25-Jul-2003)Filesystem label=OS type: LinuxBlock size=4096 (log=2)Fragment size=4096 (log=2)2097152 inodes, 4194272 blocks209713 blocks (5.00%) reserved for the super userFirst data block=0128 block groups32768 blocks per group, 32768 fragments per group16384 inodes per groupSuperblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 4096000Writing inode tables: doneCreating journal (8192 blocks): doneWriting superblocks and filesystem accounting information: doneThis filesystem will be automatically checked every 34 mounts or180 days, whichever comes first. Use tune2fs -c or -i to override.You have new mail in /var/spool/mail/root我們嘗試向RAID中寫入一個test2文件：rootlocalhost eric4ever# vi test2copy succeed!erictlfurl/urldone!rootlocalhost eric4ever# lsLATEST.tgz mdadm-2.6.2 test2rootlocalhost eric4ever# mount /dev/md0 /mnt/md0rootlocalhost eric4ever# df -lhFilesystem Size Used Avail Use% Mounted on/dev/sda1 2.9G 1.8G 1.1G 63% /dev/sda3 4.6G 33M 4.3G 1% /optnone 125M 0 125M 0% /dev/shm/dev/md0 16G 33M 15G 1% /mnt/md0rootlocalhost eric4ever# ls /mnt/md0lost+foundrootlocalhost eric4ever# cp ./test2 /mnt/md0rootlocalhost eric4ever# ls /mnt/md0lost+found test2rootlocalhost eric4ever# ls -lh /mnt/md0total 20Kdrwx- 2 root root 16K May 24 11:55 lost+found-rw-r-r- 1 root root 63 May 24 11:56 test2使用mdadm -detail /dev/md0(或mdadm -D /dev/md0)命令以及cat /proc/mdstat命令可以查看RAID設備的狀態(tài)：rootlocalhost eric4ever# mdadm -D /dev/md0 (或mdadm -detail /dev/md0)/dev/md0: Version : 00.90.00 Creation Time : Thu May 24 13:45:35 2007 Raid Level : raid5 Array Size : 16777088 (16.00 GiB 17.18 GB) Used Dev Size : 8388544 (8.00 GiB 8.59 GB) Raid Devices : 3 Total Devices : 5Preferred Minor : 0 Persistence : Superblock is persistent Update Time : Thu May 24 13:45:36 2007 State : active, degraded, recoveringActive Devices : 2Working Devices : 4Failed Devices : 1 Spare Devices : 2 Layout : left-symmetric Chunk Size : 64KRebuild Status : 16% complete UUID : 4b15050e:7d0c477d:98ed7d00:0f3c29e4 Events : 0.2 Number Major Minor RaidDevice State 0 8 16 0 active sync /dev/sdb 1 8 32 1 active sync /dev/sdc 2 0 0 2 removed 3 8 64 3 spare /dev/sde 4 8 48 4 spare /dev/sdd通過mdadm -D命令，我們可以查看RAID的版本、創(chuàng)建的時間、RAID級別、陣列容量、可用空間、設備數(shù)量、超級塊、更新時間、各個設備的狀態(tài)、RAID算法以及塊大小等信息，通過上面的信息我們可以看到目前RAID正處于重建過程之中，進度為16%，其中/dev/sdb和/dev/sdc兩塊盤已經(jīng)同步。使用watch命令每個30秒刷新一下查看的進度：rootlocalhost eric4ever# watch -n 30 cat /proc/mdstatEvery 30s: cat /proc/mdstat Thu May 24 13:55:56 2007Personalities : raid5read_ahead 1024 sectorsmd0 : active raid5 sdd4 sde3 sdc1 sdb0 16777088 blocks level 5, 64k chunk, algorithm 2 3/2 UU_ =. recovery = 72.3% (6067444/8388544) finish=3.7minspeed=10324K/secunused devices: 當進度進行到100%時，顯示如下：Every 30s: cat /proc/mdstat Thu May 24 14:00:57 2007Personalities : raid5read_ahead 1024 sectorsmd0 : active raid5 sdd2 sde3 sdc1 sdb0 16777088 blocks level 5, 64k chunk, algorithm 2 3/3 UUUunused devices: 這是我們再使用mdadm -D命令查看一下：rootlocalhost eric4ever# mdadm -D /dev/md0 (或mdadm -detail /dev/md0)/dev/md0: Version : 00.90.00 Creation Time : Thu May 24 13:45:35 2007 Raid Level : raid5 Array Size : 16777088 (16.00 GiB 17.18 GB) Used Dev Size : 8388544 (8.00 GiB 8.59 GB) Raid Devices : 3 Total Devices : 5Preferred Minor : 0 Persistence : Superblock is persistent Update Time : Thu May 24 13:59:48 2007 State : activeActive Devices : 3Working Devices : 4Failed Devices : 1 Spare Devices : 1 Layout : left-symmetric Chunk Size : 64K UUID : 4b15050e:7d0c477d:98ed7d00:0f3c29e4 Events : 0.3 Number Major Minor RaidDevice State 0 8 16 0 active sync /dev/sdb 1 8 32 1 active sync /dev/sdc 2 8 48 2 active sync /dev/sdd 3 8 64 3 spare /dev/sde這時我們可以發(fā)現(xiàn)sdb、sdc、sdd三塊盤均已經(jīng)同步。一般來說，一個新陣列被創(chuàng)建后我們最好創(chuàng)建一個/etc/mdadm.conf文件。沒有該文件在激活陣列時我們就得指定更詳細的信息，為方便，我們使用下列命令：rootlocalhost eric4ever# mdadm -detail -scanARRAY /dev/md0 level=raid5 num-devices=3 spares=1 UUID=4b15050e:7d0c477d:98ed7d00:0f3c29e4rootlocalhost eric4ever# mdadm -detail -scan /etc/mdadm.confrootlocalhost eric4ever# cat /etc/mdadm.confARRAY /dev/md0 level=raid5 num-devices=3 spares=1 UUID=4b15050e:7d0c477d:98ed7d00:0f3c29e4啟動停止RAID使用-stop或-S命令選項可以停止運行的陣列(注意： 停止前必須先umount)：rootlocalhost eric4ever# umount /mnt/md0rootlocalhost eric4ever# mdadm -S /dev/md0 (或mdadm -stop /dev/md0)mdadm: stopped /dev/md0重新啟動可以使用：rootlocalhost eric4ever# mdadm -As /dev/md0mdadm: /dev/md0 has been started with 3 drives and 1 spare.注意：停止之前需要umount，然后mdadm S /dev/md0 啟動：rootcentos /# mdadm -A /dev/md0 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1 如果已經(jīng)建立了配置文件，啟動可使用命令mdadm As /dev/md0 為使系統(tǒng)reboot后能夠自動掛載raid目錄，需要修改/etc/fstab文件，加入如下一行：/dev/md0 /mnt/md0 ext3 defaults 0 0附：簡單的故障模擬四、故障測試 1.測試前，在/mnt/radi下建立一個文件使用 rootcentos raid# ll / |tee ./ls.txt mdadm自帶命令可以標記某塊硬盤為損壞，參看幫助mdadm -manage h。 故障測試中，標記sda硬盤損壞 rootcentos /# mdadm /dev/md0 -f /dev/sda1 mdadm: set /dev/sda1 faulty in /dev/md0 2.查看狀態(tài) rootcentos /# more /proc/mdstat Personalities : raid5 md0 : active raid5 sdd13 sdc12 sdb11 sda14(F) 16771584 blocks level 5, 64k chunk, algorithm 2 3/2 _UU =. recovery = 22.8% (1916928/8385792) finish=1.9min speed=55892K/sec unused devices: rootcentos /# 因為sdd硬盤為熱備，系統(tǒng)顯示raid重構中，進度22.8%，且sda1標記（F）。 稍等再次查看： rootcentos/# more /proc/mdstat Personalities : raid5 md0 : active raid5 sdd10 sdc12 sdb11 sda13(F) 16771584 blocks level 5, 64k chunk, algorithm 2 3/3 UUU unused devices: rootcentos /# mdadm -D /dev/md0 /dev/md0: Version : 00.90.01 Creation Time : Fri Jun 1 13:00:31 2007 Raid Level : raid5 Array Size : 16771584 (15.99 GiB 17.17 GB) Device Size : 8385792 (7.100 GiB 8.59 GB) Raid Devices : 3 Total Devices : 4 Preferred Minor : 0 Persistence : Superblock is persistent Update Time : Fri Jun 1 13:23:49 2007 State : clean Active Devices : 3 Working Devices : 3 Failed Devices : 1 Spare Devices : 0 Layout : left-symmetric Chunk Size : 64K Number Major Minor RaidDevice State 0 8 49 0 active sync /dev/sdd1 1 8 17 1 active sync /dev/sdb1 2 8 33 2 active sync /dev/sdc1 3 8 1 -1 faulty /dev/sda1 UUID : 63cb965b:79486986:d389c551:67677f20 Events : 0.16 rootcentos /# 3.移除損壞的硬盤 rootcentos /# mdadm /dev/md0 -r /dev/sda mdadm: hot removed /dev/sda rootcentos /# 至此可以拿下?lián)p壞的硬盤 4.添加新的硬盤到raid中 新硬盤接入系統(tǒng)中，應進行正確的分區(qū)，且最好與之前換掉的硬盤保持相同的設備號。然后執(zhí)行命令 rootcentos /# mdadm /dev/md0 -a /dev/sda 模擬故障同raidtools一樣，mdadm也可以軟件模擬故障，命令選項為-fail或-set-faulty：rootlocalhost eric4ever# mdadm -set-faulty -helpUsage: mdadm arraydevice options component devices.This usage is for managing the component devices within an array.The -manage option is not needed and is assumed if the first argumentis a device name or a management option.The first device listed will be taken to be an md array device, andsubsequent devices are (potential) components of that array.Options that are valid with management mode are: -add -a : hotadd subsequent devices to the array -remove -r : remove subsequent devices, which must not be active -fail -f : mark subsequent devices a faulty -set-faulty : same as -fail -run -R : start a partially built array -stop -S : deactivate array, releasing all resources -readonly -o : mark array as readonly -readwrite -w : mark array as readwriterootlocalhost eric4ever# mdadm -fail -helpUsage: mdadm arraydevice options component devices.This usage is for managing the component devices within an array.The -manage option is not needed and is assumed if the first argumentis a device name or a management option.The first device listed will be taken to be an md array device, andsubsequent devices are (potential) components of that array.Options that are valid with management mode are: -add -a : hotadd subsequent devices to the array -remove -r : remove subsequent devices, which must not be active -fail -f : mark subsequent devices a faulty -set-faulty : same as -fail -run -R : start a partially built array -stop -S : deactivate array, releasing all resources -readonly -o : mark array as readonly -readwrite -w : mark array as readwrite接下來我們模擬/dev/sdb故障：rootlocalhost eric4ever# mdadm -manage -set-faulty /dev/md0 /dev/sdbmdadm: set /dev/sdb faulty in /dev/md0查看一下系統(tǒng)日志，如果你配置了冗余磁盤，可能會顯示如下信息： kernel: raid5: Disk failure on sdb, disabling device. kernel: md0: resyncing spare disk sde to replace failed disk檢查/proc/mdstat，如果配置的冗余磁盤可用，陣列可能已經(jīng)開始重建。首先我們使用mdadm -detail /dev/md0命令來查看一下RAID的狀態(tài)：rootlocalhost eric4ever# mdadm -detail /dev/md0/dev/md0: Version : 00.90.00 Creation Time : Thu May 24 13:45:35 2007 Raid Level : raid5 Array Size : 16777088 (16.00 GiB 17.18 GB) Used Dev Size : 8388544 (8.00 GiB 8.59 GB) Raid Devices : 3 Total Devices : 5Preferred Minor : 0 Persistence : Superblock is persistent Update Time : Thu May 24 14:07:55 2007 State : active, degraded, recoveringActive Devices : 2Working Devices : 3Failed Devices : 2 Spare Devices : 1 Layout : left-symmetric Chunk Size : 64KRebuild Status : 3% complete UUID : 4b15050e:7d0c477d:98ed7d00:0f3c29e4 Events : 0.6 Number Major Minor RaidDevice State 0 8 16 0 faulty spare /dev/sdb 1 8 32 1 active sync /dev/sdc 2 8 48 2 active sync /dev/sdd 3 8 64 3 spare rebuilding /dev/sde查看/proc/mdstat：rootlocalhost eric4ever# cat /proc/mdstatPersonalities : raid5read_ahead 1024 sectorsmd0 : active raid5 sdb4 sde3 sdd2 sdc1 16777088 blocks level 5, 64k chunk, algorithm 2 3/2 _UU =. recovery = 10.2% (858824/8388544) finish=12.4min speed=10076K/secunused devices: 再查看一下RAID狀態(tài)：rootlocalhost eric4ever# mdadm -detail /dev/md0/dev/md0: Version : 00.90.00 Creation Time : Thu May 24 13:45:35 2007 Raid Level : raid5 Array Size : 16777088 (16.00 GiB 17.18 GB) Used Dev Size : 8388544 (8.00 GiB 8.59 GB) Raid Devices : 3 Total Devices : 5Preferred Minor : 0 Persistence : Superblock is persistent Update Time : Thu May 24 14:08:27 2007 State : active, degraded, recoveringActive Devices : 2Working Devices : 4Failed Devices : 1 Spare Devices : 2 Layout : left-symmetric Chunk Size : 64KRebuild Status : 11% complete UUID : 4b15050e:7d0c477d:98ed7d00:0f3c29e4 Events : 0.8 Number Major Minor RaidDevice State 0 0 0 0 removed 1 8 32 1 active sync /dev/sdc 2 8 48 2 active sync /dev/sdd 3 8 64 3 spare /dev/sde 4 8 16 4 spare /dev/sdb已經(jīng)完成到11%了。查看一下日志消息：rootlocalhost eric4ever# tail /var/log/messagesMay 24 14:08:27 localhost kernel: - rd:3 wd:2 fd:1May 24 14:08:27 localhost kernel: disk 0, s:0, o:0, n:0 rd:0 us:0 dev:dev 00:00May 24 14:08:27 localhost kernel: disk 1, s:0, o:1, n:1 rd:1 us:1 dev:sdcMay 24 14:08:27 localhost kernel: disk 2, s:0, o:1, n:2 rd:2 us:1 dev:sddMay 24 14:08:27 localhost kernel: RAID5 conf printout:May 24 14:08:27 localhost kernel: - rd:3 wd:2 fd:1May 24 14:08:27 localhost kernel: disk 0, s:0, o:0, n:0 rd:0 us:0 dev:de