domino故障分析及處理方法by ibm

SoftwareDevelopmentDomino故障分析及處理方法IBMLotus?2007IBMIBMSoftwareDevelopmentDominoDomino IBMSoftwareDevelopment配置Domino配置自動診斷收集配置故障分析 IBMSoftwareDevelopment配置基本選項 IBMSoftwareDevelopment配置基本選項 IBMSoftwareDevelopment配置自動診斷收集 IBMSoftwareDevelopment配置自動診斷收集（ IBMSoftwareDevelopment使用Faultyzer數(shù)據(jù)庫 IBMSoftwareDevelopment IBMSoftwareDevelopmentDomino故障恢復概念NSD及組成故障數(shù)據(jù)收集和分析方法案例 IBMSoftwareDevelopmentNSDNSD(NotesSystemDiagnostic)是LotusDomino產(chǎn)品主要的FFDC診斷工具之一。LotusDomino產(chǎn)品主要包括：FFDC=FirstFailureData可用于分析Notes/Domino及其相關產(chǎn)品的問題，如：NSDNotes客戶端也同樣適用 IBMSoftwareDevelopmentNSD可以在所有平臺上使用(Mac除外ND6-iSeriesNSDND7-iSeriesNSD6.5.4,iSeries的NSD也開始包含Memcheck（NSD_RUN_MEMCHECK=1來設置Unix平臺–NSD是一個 W32–NSD是一個可執(zhí)行文件 IBMSoftwareDevelopmentNSDNSD可以兩種方式運行(掛起(宕機)–nsd.exe/nsd.shLotusNSD時如果不帶參數(shù)，NSDND6-“nsd_all_<ND7 SoftwareDevelopmentNSD基本信息－W32W32NSD綁定到ND(例如，記錄堆棧信息時)，會一直保持綁定狀態(tài):NSDnsdW2000NSD會殺死所有DominoW2003XP平臺，NSD結束之后自動與Domino手動運行NSD時，把運行NSD的DOS IBMSoftwareDevelopmentNSD基本信息－W32NSDnsd–detach(NSD與進程分離而不用殺死進程-XP&nsd–info(只收集系統(tǒng)信息nsd–noinfo(除了系統(tǒng)信息，其他都收集nsdmemcheck(memcheck信息

=>=>=>nsd–nomemcheck(除了memcheck信息，其他都收集=>nsd–kill(殺死所有notes進程NSDdump(收集堆棧信息quit-f(–會退出所有Domino進程

=> IBMSoftwareDevelopmentNSD基本信息－UnixNSD綁定到Domino手動運行NSD時，需要與啟動Domino IBMSoftwareDevelopmentNSD基本信息－UnixNSDnsd–batch(運行NSD但是不將輸出打印到主控臺nsd–info(只收集系統(tǒng)信息)nsd–noinfo(除了系統(tǒng)信息，其他都收集nsd–memcheck(memcheck信息)nsd–nomemcheck(除了memcheck信息，其他都收集

=>=>nsd–kill(殺死所有notes進程nsd–ps(顯示運行的進程nsd–lsof(顯示打開的Notes文件nsd–nolsof(除了打開文件的信息，其他都收集nsd–user(指定特定的用戶NSD

=>=>=>=> IBMSoftwareDevelopmentNSD基本信息－iSeriesNSDiSeries上是一個不同的工具NSD包括NSD沒有參數(shù)，只有故障恢復的時候才會被調用DMPSVRSTKS(R6.5.4及以前，是一個獨立的工具 IBMSoftwareDevelopmentNSD(堆棧調用Memcheck(Domino內存對象 IBMSoftwareDevelopment@@@@@@@@@@@@@@@@@ProcessTableINFOPID 0???0[0[000[0[00 -[-[- *[ IBMSoftwareDevelopment進程前有一個‘->’，NSD進程前有一個‘*’的，表明該進程不是在當前實例下運行的Notes進程NSDNSD的任務（比如宕機時）或從哪個數(shù)據(jù)路徑下運行的（手工運行時）Domino/NotesNotesAPI的任何進程，包括第的代碼 IBMSoftwareDevelopmentCurrentFSC TIMEA100Apr-0:11Z110:00A0 -0:00A10 -0:00A14-40:45/usr/sbin/syncdA0-0:00A0-0:00A0-0:03A10 -0:00A0-0:02sendmail:A10-0:00A0Apr-0:00A0-0:00A0 Apr-5:12A0-0:00Y3400010-/usr/bin/shA0Apr-A0Apr-/usr/sbin/biod 6020 Apr -0:28A0-/usr/sbin/nfsdA0-A0-A0-A0-grep340001Aglapps0112750 6020e8f7d Apr -269:27ibmpow/amgr-e240001 root

A08-AA08-A0c-A-IBMSoftwareDevelopment進程信息－Notes->Instance#1:IniFile DataDir:e:\notes\dataProgDir:e:\notesProcList:e:\notes\nlogasio.EXE(77c)e:\notes\nevent.EXE(798)e:\notes\nReplica.EXE(af8)e:\notes\nUpdate.EXE(b04)e:\notes\nUpdate.EXE(b0c)e:\notes\nAMgr.EXE(b18)e:\notes\nAdminp.EXE(b24)e:\notes\nCollect.EXE(b48)e:\notes\nIMAP.EXE(b54)e:\notes\nPOP3.EXE(b60)e:\notes\nHTTP.EXE(b6c)e:\notes\nSMTP.EXE(b74)e:\notes\nAMgr.EXE(c7c)e:\notes\nAMgr.EXE(c84) IBMSoftwareDevelopment進程信息－Notes@@@@@@@@@@@@@CurrentNotesprocesses@@@@@@@@@@@@@Fri 806:26:18CDTStatusis:D

PROCESS--processis--processisT/status--processterminatedwithexitstatusS/signal--processkilledwithsignal --Unknownusername

pid0?.1-R..26860R...R...R...R...R...R...R... -4 -3 -2 -1R...R...R... IBMSoftwareDevelopment為所有Domino為分析宕機/W32如果是宕機線程，會分成兩個部分UNIX IBMSoftwareDevelopment由于堆棧調用只顯示當前的情況，它并不能記錄如何導致這種情況的過程 IBMSoftwareDevelopmentW32關鍵字Fatal/W32的異常線程的堆棧調用由兩部分組成 IBMSoftwareDevelopmentW32###FATALTHREAD37/81 ###FP=0x0c55fe40,PC=0x60007dc2,SP=0x0c55fe24,stksize=28###EAX=0x0c55fe3c,EBX=0x ,ECX=0x ,EDX=0x###ESI=0x528199b8,EDI=0x ,CS=0x b,SS=0x### , , b, Exceptioncode:c @[1]0x60007dc2 Z@8+50@[2]0x601a00c1nnotes._OSLocalAlloc@8+17 @[3] fnhttpstack.HTDomWSRequest::InitDomWSRequest+31 @[4]0x1000f5a6nhttpstack.HTRequestExtContainer::HTRequestExtContainer+1020x1001b1f9nhttpstack.HTRequest::HTRequest+1210x100229f2nhttpstack.HTSession::InitRequest+162 fnhttpstack.HTSession::StartRequest+63 0x1002a99dnhttpstack.HTWorkerThread::CheckForWork+2850x1002a827nhttpstack.HTWorkerThread::ThreadMain+87nnotes._ThreadWrapper@4+212 IBMSoftwareDevelopmentW32Exceptioncode:

#Topofthe#0c55fe24 #c @[1]0x60007dc2 #0c55fe400c55fe54 # f

10032de90c55fe98

IBMSoftwareDevelopmentUNIXUNIXFatal/Panic需要關注Fatal/Panic SoftwareDevelopmentUNIX堆棧調用：AIX##thread

7/10::diioppid=13252,k-id=59465,##stack ::k-state=wait,stkmax-size=262144,ptrgl._ptrgl()at

cur-raise.nsleep(??,??)raise.nsleep(??,??)atsleep(??)at

OSRunExternalScript(??)at0xd0f3b26cOSFaultCleanup(??,??,??)at0xd0f3c228fatal_error(??,??,??)at0xd1e553bcstrlen()at0xd01d95e0Cstrlen(??)at0xd0d4a9bcANNSearchItemByName6ANNoteFPUcUc(??,??,??)atANNFindOrCreateItem6ANNoteFPUcUc(??,??,??)atANNFindItem6ANNoteFPUc(??,??)at0xd382e2a8ANNReceItem6ANNoteFPUcPUsUsUlN21(??,??,

??,??,??,??)at ceItem6ANNoteFPUcPUsUsUl(??,??,??,??,??)..

IBMSoftwareDevelopmentUNIX堆棧調用：Solaris######thread50/61::http,pid=11903,lwp=50,tid=50######ff2195acnanosleep(de07b1b8,ff07e230 (1,de07b220,40,100,de07b337,fc800000)+fd9fe978OSRunExternalScript(0,1393d8c,11,26c00,fed925ac,26d40)+fd9fd68cOSFaultCleanup(0,0,125800,2e7f,1393a28,fed925ac)+fd9dac18fatal_error(b,de07bd20,de07ba68,0,0,0)+f83c79641cCosHSolarisPchained_handler(1,fd9daa74,f853ce2c,de07ba68)+f820a7acJVM_handle_solaris_signal(0,2b0708,de07ba68,f84c8000,b,de07bd20)+ff085fecsighndlr(b,de07bd20,de07ba68,f820a8cc,0,0)+ff07fdd8call_user_handler(b,de07bd20,de07ba68,0,0,0)+ ff07ff88sigacthandler(b,de07bd20,de07ba68,0,ab6c80,13a6814)+---calledfromsignalhandlerwithsignal11(SIGSEGV)--fd2fcc581cJURLTargetJGetDbFile6M_pc_(de07f27c,e, ,0,de07c0ac,3)+fd2d94481cFHaikuDCtxQGetFormsCachePtr6F_pnMHuFormsCache(de07bff4,7e68,de07cc0c)+fd2e7f181cFHaikuHGetForm6FrnHSafePtr4nGHuForm(de07cc08,dfff80,de07cc0c)+fd28c7a81cOCustomResponseQAttemptToProcess(de07d028,d,fd639028,fd63900c)+... IBMSoftwareDevelopmentUNIX堆棧調用：Linux-----Thread12759---- :nanosleep+0x11(1,41f9ac54,3150,1,4bc08608,0)+0x406a2421:OSRunExternalScript+0x15d(0,41f9ac54,4bc08a1c,4bc08a1c, 0x406a14e2:OSFaultCleanup+0x4b2(0,0,0,8141f6c,6,4bc08cc8)+20c0x40682d35:fatal_error+0x12d(6,4bc08cc8,4bc08d5c,4bc08b68,0x494ee8a1:sysSignalCatchHandler+0x5d(6,4bc08cc8,4bc08d5c, ,4bc08d5c,6)0x494ef12c:userSignalHandler+0x68(6,4bc08cc8,4bc08d5c,494ee844, ,4bc08d5c)+280x494ef0ba:intrDispatch+0xba(6,4bc08cc8,4bc08d5c, ,40043afc,0)+100x494ef31c:intrDispatD+0x60(6,4bc08cc8,4bc08d48,4281db00,31d7,31d7) :abort+0x199(428e4d8c)0x4274a72a:terminateFv+0x1a(428e4d8c,4bc12070,0,0,0,0)+

0x40d93f18:ExtensionProc 0x40d61a7a:Execute13ExtensionProc+0x12a(7c0c61cc,41f9ac54,7c0c61cc,7c0c6120) IBMSoftwareDevelopmentJOB:085347/QNOTES/SERVER THREAD:0xf_CXX_PEP_C_pemainServerMainMainServerInitInitializeNSFRm

8QP0ZPCP2 MAIN8 1 1NIFDATA4 _RmDispatchFR11MemGrowable

ExceptionOccurreOS400DominoCancelHandle LIBMAIN fatal_error1OSFaultClesystem2sys6QC2SYS _C_pe m _C_pe m IBMSoftwareDevelopmentDomino IBMSoftwareDevelopmentMemcheckResourceUsage IBMSoftwareDevelopmentSharedSummaryofShared關鍵字"Top“SharedOSField”（<=R6.5.4）MM/關鍵字"Open IBMSoftwareDevelopmentSharedMemory續(xù)關鍵字"NIFCollection關鍵字"ServerTask關鍵字"Network IBMSoftwareDevelopmentSharedMemoryPool : OVERHEAD%used% R6.5.5or<@@------NotesMemoryyzer(memcheck)->SharedMemoryStats(Time :

98%98% IBMSoftwareDevelopmentTop#------Top10SharedMemoryBlockBY BYHANDLEHandles||||5||4|1|||| IBMSoftwareDevelopmentSharedOSField(R6.5.4及以前------SharedOSFields------StartTime=04/30/200404:19:51PMCrashTime=04/30/200404:20:12ErrorMessage=PANIC:LookupHandle:handleoutofSharedDPoolSize=FaultRecovery=0xThread[ causedStaticHangtobe

ConfigFileSem=(SEM:#0:0x010d)n=0,wcnt=-1,Users=-1, =( IBMSoftwareDevelopmentMM/OSField(R6.5.5及以后<@@------NotesMemoryyzer(memcheck)->MM/OSStructureInformation(Time08:13:20) StartTime=13/10/200708:39:22PMCrashTime=15/10/200708:12:37AMConsoleLogEnabled=1ConsolePosition=ErrorMessage=PANIC:LookupHandle:handlenotallocatedConsolePosition=331SharedDPoolSize=CrashLimits=3crashesin5StaticHang=[ nhttp:0d48:001f]/[ nhttp:0d48:0944](0xd48/0x1f/ConfigFileSem=(SEM:#0:0x010d)n=0,wcnt=-1,Users=-1,Owner= :0000] =(RWSEM:#52:0x410f) t=0Writer=[ /4 : IBMSoftwareDevelopmentResourceUsage(一站式服務OpenOpenOpenOpen IBMSoftwareDevelopmentResourcesPer**VThread[ .MappedTo:PThread[ndiiop:0904: using:PrimalThread[ SOBJ:addr=0x4cd7261c, SOBJ:addr=0x51cd039c, SOBJ:addr=0x4d4aba64,..Database:

3401,By:CN=Benny3664,By:CN=Benny view:hCol=3666,cg=N,noteID=798,(archiveLookup)| 3665,By:CN=Juergen file:fd:2388, IBMSoftwareDevelopment系統(tǒng)信息LocalKernel IBMSoftwareDevelopmentResourceLimitsResourcetime(seconds)unlimitedfile(blocks)unlimiteddata(kbytes)unlimitedstack(kbytes)8192nofiles(descriptors)256memory(kbytes)unlimitedHardtime(seconds)unlimitedfile(blocks)unlimiteddata(kbytes)unlimitedstack(kbytes)unlimitedmemory(kbytes)unlimited IBMSoftwareDevelopment系統(tǒng)信息LocalNetwork IBMSoftwareDevelopmentPerformanceTotalPhysicalMemory:2.0GAvailPhysicalMemory:985.9MMemoryUsage :72%TotalPagingFile :-1AvailPagingFile :-1TotalVirtualAvailVirtual

... IBMSoftwareDevelopment程序/數(shù)據(jù)路徑 列 IBMSoftwareDevelopmentNSD來 IBMSoftwareDevelopmentNSD如此重要如果問題比較復雜，一套堆棧調用信息往往所有 IBMSoftwareDevelopmentCrash1:May###FATALTHREAD3/73[nSERVER:0b14:0618]###FP=050fdf00,PC=605d22f0,SP=050fdefc,stksize=4Exceptioncode:c )@[2]0x600078fc@[3]0x60007cf3nnot @[4]0x601a3001nnotes._OSLocalAlloc@8+17@[6]0x )@[7]0x60864aec@[8]0x60088e64nnotes._NAMELookupDiskLookup@20(6,50fff00,50fff50,50fff58)@[9]0x600a448fnnotes._Lookup@32(200,1,1224d28,1)@[10]0x6087a099 [12]0x7c57438b IBMSoftwareDevelopmentCrash2:May###FATALTHREAD3/72[nSERVER:0b38:06bc]###FP=0520eeb4,PC=6019ea58,SP=0520ee7c,stksize=56Exceptioncode:c @[2]0x600084b0nnotes._OSLockWriteFRWSem@4+16 b,200191f)@[3]0x608455bcnnotes._hosRequestMutexSem+12(200191f,0,2000f05,b @[4]0x bnnotes._sqlpgrsp+43(2000f05,2,b @[5]0x607d307annotes._hlgReleaseLogSpace+122@[6]0x607e616dnnotes._RmLogChange@12+141 ,1,520f11c)@[7]0x607dd738 mit@8+696(0,0,0,520f154)(520f154,520f11c,3e7a0e8,520f154)@[9]0x600040a6@[10]0x bnnotes._UnlockDbRO@8+27 ,520f1bc,520f7a0)@[12]0x600202a5

IBMSoftwareDevelopmentCrash3:June###FATALTHREAD2/72[nSERVER:0364:0560]###FP=031ddfe8,PC= b,SP=031ddfdc,stksize=12Exceptioncode:c @[1]0x @[2]0x )@[3]0x60864aecnnotes._ReturnLNABNameInfo@12(1db,31dfb4c,31df90c, @[4]0x60088e64nnotes._NAMELookupDiskLookup@20(6,31dff00,31dff50,31dff58)@[5]0x600a448fnnotes._Lookup@32(200,1,120641c,1)@[6]0x6087a099nnotes._NAMELookupThread@4+505(29605c0,0,0,0)@[7]0x6011df84nnotes._ThreadWrapper@4+212[8]0x7c57438b IBMSoftwareDevelopmentNSD多種因素會影響NSD IBMSoftwareDevelopmentNSD大小和收集需要的時間(續(xù)相同的因素會影響NSD只啟用必要的任務W32手工運行的時候，最小化命令窗口有助于縮短運行 IBMSoftwareDevelopmentNSDNSD所提供的信息會幫助我們往正確的consolememoryOS關于NSD IBMSoftwareDevelopmentDomino故障恢復概念NSD及組成故障數(shù)據(jù)收集和分析方法案例 IBMSoftwareDevelopmentDomino SoftwareDevelopment一個定義明確的 ：服務器在程序或操作系統(tǒng)級別碰到了一個致命性的異常錯誤(PANIC,FATAL,AccessViolation)如果配置了故障恢復，NSD日志會自動創(chuàng)建，并記錄相關進程的堆棧調用信息 IBMSoftwareDevelopment– IBMSoftwareDevelopment(或特定任務仍在運行，但是對于用戶沒有任何NSD IBMSoftwareDevelopment以下原因可能會導致掛起 IBMSoftwareDevelopment服務器任務從操作系統(tǒng)任務列表中，但沒有任何錯(極少出現(xiàn)這種問題(任務已經(jīng)不再運行了Domino(因為它已經(jīng)不再運行 IBMSoftwareDevelopmentW32W32也可能發(fā)現(xiàn)C+ IBMSoftwareDevelopment診斷Domino IBMSoftwareDevelopment診斷Domino所需要的數(shù)據(jù)NSD(文件名的時間信息是運行NSD的時間Consolelog（文件名的時間信息是服務器啟動的時間SEMDEBUG（文件名的時間信息是服務器啟動的時間Htthr IBMSoftwareDevelopment診斷Domino IBMSoftwareDevelopment診斷Domino宕機－處理(續(xù)如果服務器連續(xù)宕機…如果是router宕機，重命名 根據(jù)宕機的信息來決定是否要添加的調試參數(shù)來確定引起宕機的函重新啟動操作系統(tǒng)對系統(tǒng)數(shù)據(jù)庫進行數(shù)據(jù)庫操作（names.nsf,聯(lián)系800 IBMSoftwareDevelopment診斷Domino宕機－處理(續(xù) IBMSoftwareDevelopment如何利用NSDSharedOS關鍵字：sharedOpen宕機的任務是不是第應用程序 IBMSoftwareDevelopment如何利用NSD來診斷宕機的問題(續(xù) IBMSoftwareDevelopmentHTTP用戶是否能連接Domino IBMSoftwareDevelopment IBMSoftwareDevelopment診斷服務器掛起－現(xiàn)場數(shù)據(jù)收集在Domino控制臺輸入命令（每5分鐘SHSHTASKSSHTASKSSHSHLROUTERSHOWSHLAMGR IBMSoftwareDevelopment診斷服務器掛起－現(xiàn)場數(shù)據(jù)收集手動運行NSD（非iSeries平臺 多個NSD（每隔5分鐘Win2000：在NSD提示符（nsd>）輸入“dump”（再次收集堆 SoftwareDevelopment診斷服務器掛起－現(xiàn)場數(shù)據(jù)收集DMPDOMSVRC(6.5.5or servers/eserver/iseries/domino/devtools/dmp/) IBMSoftwareDevelopment診斷服務器掛起－現(xiàn)場數(shù)據(jù)收集NSD文件名的時間信息是運行NSD的時間Consolelog（文件名的時間信息是服務器啟動的時間Memorydump（文件名的時間信息是運行memorydump的時間 IBMSoftwareDevelopment診斷HTTP(或特定任務仍在運行，但是對于輸入沒有任何NSDWeb用戶不能服務Notes客戶端能正常服務 IBMSoftwareDevelopment診斷HTTP IBMSoftwareDevelopment診斷HTTP掛起－現(xiàn)場數(shù)據(jù)收集在Domino控制臺輸入命令（每SHSHTASKSLHTTPSHOWTHREADSHTASKSSHSHLROUTERSHOWSHLAMGR IBMSoftwareDevelopment診斷HTTP掛起－現(xiàn)場數(shù)據(jù)收集手動運行NSD或者DumpStacks（每所需要的數(shù)據(jù)ConsoleNSD(dumpHtthr IBMSoftwareDevelopmentDomino故障恢復概念NSD及組成故障數(shù)據(jù)收集和分析方法案例 IBMSoftwareDevelopment宕機案例–NSDNSDConsole IBMSoftwareDevelopment宕機案例1###FATALTHREAD39/63[nhttp:0d48:0944]###FP=0x1274ef34,PC=0x60176b01,### ,totalstksize=262144,used###EAX=0x ###ESI=0x1274edc8,EDI=0x20a9cb61,CS=0x b,SS=0x### , , b, Exceptioncode:c @[1]0x60176b01nnotes.Panic@4+417 @[2]0x60002c16nnotes.LockMemHandle@12+294(f @[3]0x nnotes.OSMemoryGetSize@4+21(60d31400,124c5814,850c014,0) @[6]0x @[7]0x @[8] @[12]0x @[14]0x1002cc28nhttpstack.HTWorkerThread::ThreadMain+88(95ebba4,0,6000d583,0)@[15]0x60103d20nnotes.ThreadWrapper@4+208[16]0x7c57b3bc IBMSoftwareDevelopment宕機案例1 IBMSoftwareDevelopment宕機案例1 IBMSoftwareDevelopment宕機案例NSDConsole IBMSoftwareDevelopment宕機案例2##thread4/4::routerpid=28664,k-id=51215,pthr-##stack ::k-state=wait,stkmax-size=262144,cur-size=266412ptrgl._ptrgl()atsleep(??)at0xd021fc54OSRunExternalScript(??)at0xd2d582c0OSFaultCleanup(??,??,??)at0xd2d5928cfatal_error(??,??,??)at0xd399018cGetLockedLocalPool(??)at0xd2c837d4Z(??,??)at ZFUlPv(??,??)atAlloc10MemManagerXSP8192_FUlPv(??,??,??)at0xd2e871f8TransMemAlloc7ComputeFCUl(??,??)at0xd2e86e04 11TexementCFRC11Texement(??,??,??)atOperateElements 15TextOperatePlusFR11TexementT1(??,??,??,??)at0xd3e4f2dcPlus pValue13OperationType(??,??,??)at0xd3e4a554Execute6OpPlusFv(??)at0xd3e813ecGetValue 14AtFunctionNodeFv(??)at0xd2e89178GetArgValue 14ExecutableNodeFUl(??,??)at0xd2e88e20Execute6OpPlusFv(??)at0xd3e813b0GetValue 14AtFunctionNodeFv(??)at0xd2e89178GetArgValue 14ExecutableNodeFUl(??,??)at0xd2e88e20Execute6OpPlusFv(??)at0xd3e813b0GetValue 14AtFunctionNodeFv(??)at0xd2e89178GetArgValue 14ExecutableNodeFUl(??,??)at0xd2e88e20。。。。。。。。 IBMSoftwareDevelopment宕機案例2**VThread[ .MappedTo:PThrea