丟失的ASM磁盤組

1、近日遇到一個問題，ASM的磁盤組無法掛載，之前是正常的，由于一些其他的操作，數(shù)據(jù)庫啟動失敗，當問題排除時候，發(fā)現(xiàn)在數(shù)據(jù)庫整體啟動時，掛載磁盤組的環(huán)節(jié)出現(xiàn)問題。環(huán)境介紹#硬件：Vmware ESX虛擬機OS: Red hat linux 5ASM disk是通過 asmlib掛載的這個磁盤組只有一個虛擬出的硬盤，是 /dev/sdb1.#下面是我整個分析的過程1. 首先通過ASM alert.log，發(fā)現(xiàn)如下錯誤,磁盤掛載失敗，無法找到磁盤組1. SQLalterdiskgroupDATAmount2. NOTE:cacheregisteredgroupDATAnumber=1incarn=0x

2、c28a1e2d3. NOTE:cachebeganmount(first)ofgroupDATAnumber=1incarn=0xc28a1e2d4. TueDec1118:06:5520125. ERROR:noPSTquorumingroup:required2,found06. NOTE:cachedismounting(clean)group1/0xC28A1E2D(DATA)7. NOTE:dbwrnotbeingmsgdtodismount8. NOTE:lgwrnotbeingmsgdtodismount9. NOTE:cachedismountedgroup1/0xC28A1

3、E2D(DATA)10. NOTE:cacheendingmount(fail)ofgroupDATAnumber=1incarn=0xc28a1e2d11. NOTE:cachedeletingcontextforgroupDATA1/0xc28a1e2d12. GMONdismountinggroup1at8forpid17,osid3216313. ERROR:diskgroupDATAwasnotmounted14. ORA-15032:notallalterationsperformed15. ORA-15017:diskgroupDATAcannotbemounted16. ORA

4、-15063:ASMdiscoveredaninsufficientnumberofdisksfordiskgroupDATA17. ERROR:alterdiskgroupDATAmount 2. 首先檢查ASM pfile 文件，未發(fā)現(xiàn)異常1. asm_diskgroups=DATA2. instance_type=asm3. large_pool_size=12M4. remote_login_passwordfile=EXCLUSIVE3. 嘗試通過以下命令檢查磁盤是否物理存在，是如何對應(yīng)物理設(shè)備的，發(fā)現(xiàn)查詢不到ASM磁盤1. gridlgto_test$kfoddisks=all2.

5、 3. -Nonoutput-4. 5. 6. 7. gridlgto_testpeer$cd/dev/oracleasm/disks/8. gridlgto_testdisks$ls9. 10. -Nonoutput-11. 12. 13. gridlgto_testdisks$/etc/init.d/oracleasmlistdisks14. 15. -Nonoutput- 4.但是直接檢查物理設(shè)備，/dev/sdb1是存在的，說明OS已經(jīng)識別該硬盤設(shè)備，只是ASMlib無法正常識別:1. 查詢對應(yīng)的物理硬盤2. oracleOEL $ /etc/init.d/oracleasm quer

6、ydisk -d disk1 Disk DISK1 is a valid ASM disk on device 8,173. oracleOEL $ ls -l /dev/ |grep 8|grep 17 brw-r- 1 root disk 8, 17 Oct 16 14:01 sdb14. 5. rootlgto_test#ls-lst/dev/sd*6. 0brw-r-1rootdisk8,0Dec1119:29/dev/sda7. 0brw-r-1rootdisk8,2Dec1119:29/dev/sda28. 0brw-r-1rootdisk8,16Dec1119:29/dev/sd

7、b9. 0brw-r-1rootdisk8,17Dec1119:29/dev/sdb1Thisisthemisseddiskgroup10. 0brw-r-1rootdisk8,1Dec1111:29/dev/sda15.起先是考慮是否是磁盤頭損害，導(dǎo)致無法ASMlib識別該磁盤， dump磁盤頭發(fā)現(xiàn)沒有問題.1. #od-c/dev/sdb12. 3. 0000040ORCLDISKDATADG014. 5. 7760040ORCLDISKDATADG01這里補充下，如果磁盤頭信息丟失，將會顯示如下1. 0000040ORCLDISK00000000如果顯示這個結(jié)果，需要通過以下方式rena

8、medisk，具體可以參考文檔Oracleasm Listdisks Cannot See Disks (Doc ID 392527.1)1. Usetheoracleasmrenamediskutilitytoaddanasmliblabeltothedisk:2. /etc/init.d/oracleasmrenamedisk/dev/3. Ifitfails,usethe-fswitch:4. /etc/init.d/oracleasmrenamedisk-f/dev/ 6.重啟ASMLib ，檢查是否是ASMLib 問題1. rootlgto_test#/etc/init.d/orac

9、leasmrestart2. DroppingOracleASMLibdisks:3. OK4. ShuttingdowntheOracleASMLibdriver:FAILED檢查文件系統(tǒng)oracleasm文件系統(tǒng)已經(jīng)成功掛載1. rootlgto_test#df-ha2. FilesystemSizeUsedAvailUse%Mountedon3. 4. oracleasmfs000-/dev/oracleasm7.檢查 /dev/sdb1狀態(tài)，查看是否已經(jīng)marked為ASM disk，顯示已經(jīng)標記成功1. rootlgto_test#oracleasmquerydisk/dev/sdb

10、12. Device/dev/sdb1ismarkedanASMdiskwiththelabelDATADG013. 4. rootlgto_test#/sbin/serviceoracleasmscandisks5. ScanningthesystemforOracleASMLibdisks:6. OK7. 8. rootlgto_test#/etc/init.d/oracleasmlistdisks9. -Nonoutput-8.檢查 rpm package也沒有問題1. gridlgto_test$rpm-qa|greporacleasm2. oracleasmlib-2.0.4-1.e

11、l53. oracleasm-support-2.1.7-1.el54. oracleasm-2.6.18-308.el5-2.0.5-1.el5 9. 收集 Kfed logs,沒有檢查到異常新信息。1. rootlgto_test#/oracle/ora11g/product/app/grid/bin/kfedread/dev/sdb12. 3. kfbh.endian:1;0x000:0x014. 5. kfbh.hard:130;0x001:0x826. 7. kfbh.type:1;0x002:KFBTYP_DISKHEAD8. 9. kfbh.datfmt:1;0x003:0x01

12、10. 11. kfbh.block.blk:0;0x004:T=0NUMB=0x012. 13. kfbh.block.obj:2147483648;0x008:TYPE=0x8NUMB=0x014. 15. kfbh.check:3351358462;0x00c:0xc7c1abfe16. 17. kfbh.fcn.base:0;0x010:0x0000000018. 19. kfbh.fcn.wrap:0;0x014:0x0000000020. 21. kfbh.spare1:0;0x018:0x0000000022. 23. kfbh.spare2:0;0x01c:0x00000000

13、24. 25. vstr:ORCLDISKDATADG01;0x000:length=1626. 27. kfdhdb.driver.reserved0:1096040772;0x008:0x4154414428. 29. kfdhdb.driver.reserved1:825247556;0x00c:0x3130474430. 31. kfdhdb.driver.reserved2:0;0x010:0x0000000032. 33. kfdhdb.driver.reserved3:0;0x014:0x0000000034. 35. kfdhdb.driver

14、.reserved4:0;0x018:0x0000000036. 37. kfdhdb.driver.reserved5:0;0x01c:0x0000000038. 39. pat:186646528;0x020:0x0b20000040. 41. kfdhdb.dsknum:0;0x024:0x000042. 43. kfdhdb.grptyp:1;0x026:KFDGTP_EXTERNAL44. 45. kfdhdb.hdrsts:3;0x027:KFDHDR_MEMBER46. 47. kfdhdb.dskname:DATADG01;0x028:length=848. 49. kfdhd

15、b.grpname:DATA;0x048:length=450. 51. kfdhdb.fgname:DATADG01;0x068:length=852. 53. kfdhdb.capname:;0x088:length=054. 55. kfdhdb.crestmp.hi:32977140;0x0a8:HOUR=0x14DAYS=0x7MNTH=0xcYEAR=0x7dc56. 57. kfdhdb.crestmp.lo:1642529792;0x0ac:USEC=0x0MSEC=0x1c1SECS=0x1eMINS=0x1858. 59. kfdhdb.mntstmp.hi:3297714

16、0;0x0b0:HOUR=0x14DAYS=0x7MNTH=0xcYEAR=0x7dc60. 61. kfdhdb.mntstmp.lo:1664549888;0x0b4:USEC=0x0MSEC=0x1c1SECS=0x33MINS=0x1862. 63. kfdhdb.secsize:512;0x0b8:0x020064. 65. kfdhdb.blksize:4096;0x0ba:0x100066. 67. kfdhdb.ausize:1048576;0x0bc:0x0010000068. 69. kfdhdb.mfact:113792;0x0c0:0x0001bc8070. 71. k

17、fdhdb.dsksize:204797;0x0c4:0x00031ffd72. 73. kfdhdb.pmcnt:3;0x0c8:0x0000000374. 75. kfdhdb.fstlocn:1;0x0cc:0x0000000176. 77. kfdhdb.altlocn:2;0x0d0:0x0000000278. 79. kfdhdb.f1b1locn:2;0x0d4:0x0000000280. 81. kfdhdb.redomirrors0:0;0x0d8:0x000082. 83. kfdhdb.redomirrors1:0;0x0da:0x000084. 85. kfdhdb.r

18、edomirrors2:0;0x0dc:0x000086. 87. kfdhdb.redomirrors3:0;0x0de:0x000088. 89. kfdhdb.dbcompat:168820736;0x0e0:0x0a10000090. 91. kfdhdb.grpstmp.hi:32977140;0x0e4:HOUR=0x14DAYS=0x7MNTH=0xcYEAR=0x7dc92. 93. kfdhdb.grpstmp.lo:1642390528;0x0e8:USEC=0x0MSEC=0x139SECS=0x1eMINS=0x1894. 95. kfdhdb.vfstart:0;0x

19、0ec:0x0000000096. 97. kfdhdb.vfend:0;0x0f0:0x0000000098. 99. kfdhdb.spfile:58;0x0f4:0x0000003a100. 101. kfdhdb.spfflg:1;0x0f8:0x00000001102. 103. kfdhdb.ub4spare0:0;0x0fc:0x00000000104. 105. kfdhdb.ub4spare1:0;0x100:0x00000000106. 107. kfdhdb.ub4spare2:0;0x104:0x00000000108. 109. kfdhdb.ub4spare3:0;

20、0x108:0x00000000110. 111. kfdhdb.ub4spare4:0;0x10c:0x00000000112. 113. kfdhdb.ub4spare5:0;0x110:0x00000000114. 115. kfdhdb.ub4spare6:0;0x114:0x00000000116. 117. kfdhdb.ub4spare7:0;0x118:0x00000000118. 119. kfdhdb.ub4spare8:0;0x11c:0x00000000120. 121. kfdhdb.ub4spare9:0;0x120:0x00000000122. 123. kfdh

21、db.ub4spare10:0;0x124:0x00000000124. 125. kfdhdb.ub4spare11:0;0x128:0x00000000126. 127. kfdhdb.ub4spare12:0;0x12c:0x00000000128. 129. kfdhdb.ub4spare13:0;0x130:0x00000000130. 131. kfdhdb.ub4spare14:0;0x134:0x00000000132. 133. kfdhdb.ub4spare15:0;0x138:0x00000000134. 135. kfdhdb.ub4spare16:0;0x13c:0x

22、00000000136. 137. kfdhdb.ub4spare17:0;0x140:0x00000000138. 139. kfdhdb.ub4spare18:0;0x144:0x00000000140. 141. kfdhdb.ub4spare19:0;0x148:0x00000000142. 143. kfdhdb.ub4spare20:0;0x14c:0x00000000144. 145. kfdhdb.ub4spare21:0;0x150:0x00000000146. 147. kfdhdb.ub4spare22:0;0x154:0x00000000148. 149. kfdhdb

23、.ub4spare23:0;0x158:0x00000000150. 151. kfdhdb.ub4spare24:0;0x15c:0x00000000152. 153. kfdhdb.ub4spare25:0;0x160:0x00000000154. 155. kfdhdb.ub4spare26:0;0x164:0x00000000156. 157. kfdhdb.ub4spare27:0;0x168:0x00000000158. 159. kfdhdb.ub4spare28:0;0x16c:0x00000000160. 161. kfdhdb.ub4spare29:0;0x170:0x00

24、000000162. 163. kfdhdb.ub4spare30:0;0x174:0x00000000164. 165. kfdhdb.ub4spare31:0;0x178:0x00000000166. 167. kfdhdb.ub4spare32:0;0x17c:0x00000000168. 169. kfdhdb.ub4spare33:0;0x180:0x00000000170. 171. kfdhdb.ub4spare34:0;0x184:0x00000000172. 173. kfdhdb.ub4spare35:0;0x188:0x00000000174. 175. kfdhdb.u

25、b4spare36:0;0x18c:0x00000000176. 177. kfdhdb.ub4spare37:0;0x190:0x00000000178. 179. kfdhdb.ub4spare38:0;0x194:0x00000000180. 181. kfdhdb.ub4spare39:0;0x198:0x00000000182. 183. kfdhdb.ub4spare40:0;0x19c:0x00000000184. 185. kfdhdb.ub4spare41:0;0x1a0:0x00000000186. 187. kfdhdb.ub4spare42:0;0x1a4:0x0000

26、0000188. 189. kfdhdb.ub4spare43:0;0x1a8:0x00000000190. 191. kfdhdb.ub4spare44:0;0x1ac:0x00000000192. 193. kfdhdb.ub4spare45:0;0x1b0:0x00000000194. 195. kfdhdb.ub4spare46:0;0x1b4:0x00000000196. 197. kfdhdb.ub4spare47:0;0x1b8:0x00000000198. 199. kfdhdb.ub4spare48:0;0x1bc:0x00000000200. 201. kfdhdb.ub4

27、spare49:0;0x1c0:0x00000000202. 203. kfdhdb.ub4spare50:0;0x1c4:0x00000000204. 205. kfdhdb.ub4spare51:0;0x1c8:0x00000000206. 207. kfdhdb.ub4spare52:0;0x1cc:0x00000000208. 209. kfdhdb.ub4spare53:0;0x1d0:0x00000000210. 211. kfdhdb.acdb.aba.seq:0;0x1d4:0x00000000212. 213. kfdhdb.acdb.aba.blk:0;0x1d8:0x00000000214. 215. kfdhdb.acdb.ents:0;0x1dc:0x0000216. 217. kfdhdb.acdb.ub2spare:0;0x1de:0x000010. 階段總結(jié)，通過以上的分析，得出以下總結(jié)1. ASMLib 正常2. RPM包正常3. 磁盤頭沒有損壞和丟失信息4. 該硬件已經(jīng)被系統(tǒng)正常識別當前問題就是為什么ASMLib 不能正常掃描并識別到該硬盤11. 最后在檢查文件 /etc/sysconfig/oracleasm時，發(fā)現(xiàn)問題，我們需要