H852小型機故障記錄

小型機故障記錄故障機器：主機名 H852故障時間：2007年9月3日下午2點左右故障現(xiàn)象：同一網(wǎng)段的小型機與該主機通信正常，但跨網(wǎng)段的機器無法與該主機通信初步判斷：該主機的網(wǎng)絡設置被改動，或者是網(wǎng)絡設備物理或邏輯上的變動所致處理過程：1詢問網(wǎng)絡管理員，他們最近沒有對網(wǎng)絡設備作物理或邏輯上的調(diào)整2在本機查網(wǎng)絡設置和日志通過對本機的配置進行查詢，發(fā)現(xiàn)存在如下現(xiàn)象，現(xiàn)詳細記錄如下：#hostnameH852#ifconfig-aenO:flags=4e080863,80<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,PSEG,CHAIN>enl:flags=4e080863,80<UP,BROADCAST,NOTRAILERS,RUNNINQSIMPLEX,MULTICAST,GROUPRT,64BIT,PSEG,CHAIN>loO:flags=e08084b<UP,BROADCAST,LOOPBACK,RUNNINQSIMPLEX,MULTICAST,GROUPRT,64BIT>inetnetmaskOxffOOOOOObroadcast55inet6::l/0tcp_sendspace131072tcp_recvspace131072rfcl32313#netstat-rRoutingtablesDestinationGateway FlagsRefsUseIfPMTUExpGroupsRouteTreeforProtocolFamily2(Internet):defaultUG01312547enO - -UHSb00enl - =>10/8U311028enl - -loopbackUGHS02loO - -loopbackUHSb00loO - -二>h852_serviceUHSb00enO - =>h852_serviceU5104741enO - -h852_serviceloopbackUGHS016668loO - - -loopbackUHSb0269loO - - -二>h852_serviceUHSb02enO - -UHSb00enl - -rootpts/1Apr2010:58-10:59(00:00)rootpts/1Apr2010:48-10:48(00:00)rootpts/1Apr2010:38-10:38(00:00)rootpts/1Apr2010:28-10:28(00:00)rootpts/1Apr2010:18-10:18(00:00)rootpts/1Apr2010:08-10:08(00:00)rootpts/1Apr2009:58-09:58(00:00)rootpts/1Apr2009:47-09:47(00:00)rootpts/1Apr2009:37-09:37(00:00)rootpts/1Apr2009:27-09:27(00:00)rootpts/1Apr2009:17-09:17(00:00)rootpts/1Apr2009:07-09:07(00:00)rootpts/1Apr2008:57-08:57(00:00)rootpts/1Apr2008:46-08:46(00:00)rootpts/1Apr2008:36-08:36(00:00)rootpts/1Apr2008:26-08:26(00:00)rootpts/1Apr2008:16-08:16(00:00)rootpts/1Apr2008:06-08:06(00:00)rootpts/1Apr2007:56-07:56(00:00)rootpts/1Apr2007:45-07:45(00:00)rootpts/1Apr2007:35-07:35(00:00)rootpts/1Apr2007:25-07:25(00:00)通過對以上信息的分析，問題在于網(wǎng)卡enl的IP被人為改動所致在以上出錯信息中，9月3日的報錯是因為重啟HACMP的緣故，不是事故的原因。查以上信息，enl的IP應該在2007年7月16日以前被改動，具體改成什么IP,不可查知，但在9月3日出現(xiàn)故障時，可查知enl的IP被改為（正常IP應是0）故障原因分析：因為enl的IP被改為與主網(wǎng)卡（enO）同一個A網(wǎng)段的地址，所以，網(wǎng)絡會發(fā)生聚合一-就是該主機會把諸如或之類所有的地址全部當成是網(wǎng)段的主機。所以，當其它網(wǎng)段（例如IP為10.20.222..29的終端）的TCP請求從到達主機后，因為網(wǎng)絡地址聚合的原理，該主機會認為從10.20.222.?29發(fā)來的請求包是來自本地網(wǎng)，所以回包信息不會發(fā)給它的網(wǎng)關,所以，到不同網(wǎng)段的TCP包也就不會被路由出去，而是在本局域網(wǎng)內(nèi)傳遞（這就是為什么跨網(wǎng)段的終端不能訪問到該主機，而同一局域網(wǎng)的小型機能訪問該主機的原因）最后，試圖用smittcpip改回enl的IP時，發(fā)現(xiàn)其IP設置是正確的（為0）,因此推測出對enl的IP的改動是通過ifconfigenlnetmaskup命令動態(tài)實現(xiàn)的。直接用ifconfig將IP改回0,網(wǎng)絡恢復正常。127/8 loopback U2114059043IoORouteTreeforProtocolFamily24(Internetv6):::1 ::1 UH0 0loO#4#en-pt#errptIDENTIFIERTIMESTAMPTCRESOURCE_NAMEDESCRIPTIONAFA8990592E96A0D97419D605294E5FF6D19271E28854E8199FA80C79D30B78E9D30B78E9D30B78E9D30B78E9D30B78E9D30B78E9D30B78E3C81E43F92E96A0D0903163807IOgrpsvcs0903163807PUtopsvcs0903163807IOtopsvcs0903163807IOtopsvcs0903143807IOtopsvcs0903143807IOgrpsvcs0903143807UShaemd0826234007TSttyl0825124307TStty10825050007TSttyl0822205307TSttyl0821200307TSttyl0818012907TSttyl0812174207TSttyl0720000007PUtopsvcs0716171507PUtopsvcsGroupServicesdaemonstartedLocaladaptermisconfigurationdetectedTopologyServicesdaemonstartedMissingorincorrectadapteraddressTopologyServicesdaemonstoppedGroupServicesdaemonstoppedSOFTWARERECEIVEROVER-RUNONINPUTRECEIVEROVER-RUNONINPUTRECEIVEROVER-RUNONINPUTRECEIVEROVER-RUNONINPUTRECEIVEROVER-RUNONINPUTRECEIVEROVER-RUNONINPUTRECEIVEROVER-RUNONINPUTLateinsendingheartbeatLocaladaptermisconfigurationdetected5#errpt-aj5294E5FFLABEL:IDENTIFIER:TS_FDJNVAL_ADDR_ST5294E5FFDate/Time:MonSep316:38:36BEIST2007SequenceNumber:95368MachineId:000C4ADF4C00NodeId:h852Class:OType:ResourceName:INFOtopsvcsDescriptionMissingorincorrectadapteraddressProbableCausesAdapterisnotconfiguredAdapterhasnoaddressAdapterhasanaddressthatisoutsidetheconfigurationUserCausesAdapterwasincorrectlyconfiguredRecommendedActionsCorrectadapterconfiguration:itsaddressmustmatchtheboot-timeaddressexpectedbyTopologyServicesFailureCausesAproblemoccurredwhenconfiguringtheadapterRecommendedActionsCorrectadapterconfiguration:itsaddressmustmatchtheboot-timeaddressexpectedbyTopologyServicesDetailDataDETECTINGMODULErsct,force_down.C,1.11,501ERRORID6zLCZG/AUwq4/w8P.SEV08 REFERENCECODEInterfacenameenlActualaddressExpectedboot-timeaddress6#errpt-aj92E96A0DLABEL: TS_MISCFG_EMIDENTIFIER: 92E96A0DDate/Time: MonSep316:38:37BEIST2007SequenceNumber:95370MachineId:000C4ADF4C00NodeId:h852Class:UType:PENDResourceName:topsvesResourceClass:NONEResourceType:NONELocation:VPD:DescriptionLocaladaptermisconfigurationdetectedProbableCausesLocaladapterassignedincorrectIPaddressInstallCausesLocaladapterassignedincorrectIPaddressRecommendedActionsCorrectadapterIPaddressIftheproblempersists,restartTopologyServicesdaemon.DetailDataDETECTINGMODULErsct,nim_control.C,1.38,4338ERRORID6BcKuG0BUwq4/2DC/SEV08 REFERENCECODEAdapterinterfacenameenlAdapteroffset1AdapterexpectedIPaddressLABEL:TS_MISCFG_EMIDENTIFIER:92E96A0DDate/Time:Date/Time:MonJul1617:15:35BEIST2007SequenceNumber:95344MachineId:000C4ADF4C00h852h852NodeId:Class:Type:ResourceName:PENDResourceClass:topsvcsNONEResourceType:Location:NONEVPD:DescriptionLocaladaptermisconfigurationdetectedProbableCausesLocaladapterassignedincorrectIPaddressInstallCausesLocaladapterassignedincon-ectIPaddressRecommendedActionsCorrectadapterIPaddressIftheproblempersists,restartTopologyServicesdaemon.DetailDataDETECTINGMODULErsct,nim_control.C,1.38,4338ERRORID6BcKuG0rQna4/yPD1SEV08 REFERENCECODEAdapterinterfacenameenlAdapteroffset1AdapterexpectedIPaddress7部分登錄信息#lastrootpts/2RAC_P5711Sep0409:31-09:44(00:13)rootpts/1RAC_P5711Sep0409:17stillloggedin.rootpts/3h85I_serviceSep0316:51-17:04(00:13)rootpts/2RAC_P5711S叩0316:47-16:58(00:11)rootpts/1h851_serviceSep0316:42-18:57(02:15)rootIftOSep0313:55-16:40(02:44)rootpts/1Aug2415:24-16:53(01:29)rootpts/1Aug2410:14-12:24(02:10)rootpts/1Aug2120:05-20:11(00:05)rootpts/1Aug2119:30-19:31(00:01)rootpts/1h851_serviceAug2115:20-16:57(01:37)rootIftOAug1416:27-16:55(00:27)rootdtremote58:0Aug1409:03-13:38(04:35)rootpts/1Aug0911:08-15:41(04:33)rootdtremote58:0Aug0910:34-10:43(00:09)rootpts/OAug0308:59-13:16(04:17)rootpts/1h851_serviceJul1816:34-16:36(00:01)rootpts/1Jul1716:47-23:18(06:31)rootpts/1Jul1708:41-14:18(05:37)rootpts/1Jul1618:00-18:02(00:02)rootdtremote1:1Jul1616:58-17:19(00:20)rootpts/1Jul1616:48-16:55(00:07)rootrootIftOpts/OJul1616:28-17:03(11+00:34)Jul1616:25-16:26(00:00)rootpts/OMay2209:28-09:29(00:00)rootpts/OMay1615:43-15:43(00:00)rootpts/OApr2623:26-23:28(00:02)rootpts/OApr2610:04-10:08(00:04)rootpts/OApr2509:51-11:31(01:39)rootrebootshutdownrootrootIftOpts/OIftOpts/Oh851_stbyApr0610:14-16:19(1014-06:05)Apr0610:09Apr0610:00Apr0609:45-10:00(00:14)Aug2511:05-11:05(00:00)rootpts/Oh851_stbyAug2115:07-15:24(00:17)rootpts/Oh851_stbyAug2110:53-11:00(00:06)rootrebootshutdownrootrebootrebootshutdownrootrebootshutdownoracleIftOIftOIftOIftOIftOIftOftpJul2519:41-10:15(203+14:33)Jul2519:37Jul2514:56Jun2211:42-14:55(33+03:13)Jun2112:29Apr2810:54Apr2810:28Apr2810:27-10:28(00:01)Apr2810:23Apr2809:56Apr2708:58-08:58(00:00)nlastn1432lines,108241charactersshutdownIftOApr2809:56Apr2708:58-08:58(00:00)rootpts/2Apr2610:48-10:53(00:05)oracleftpApr2609:45-09:45(00:00)oracleftpApr2609:32-09:32(00:00)oracleftpApr2608:58-08:58(00:00)oracleftpApr2509:01-09:02(00:01)oracle即Apr2408:59-09:00(00:01)oraclepts/1Apr2115:36-15:48(00:11)rootpts/1Apr2110:01-10:01(00:00)rootpts/1Apr2109:51-09:51(00:00)rootpts/1Apr2109:40-09:41(00:00)rootpts/1Apr2109:30-09:30(00:00)rootpts/1Apr2109:20-09:20(00:00)rootpts/1Apr2109:10-09:10(00:00)rootpts/1Apr2109:00-09:00(00:00)rootpts/1Apr2108:50-08:50(00:00)rootpts/1Apr2108:39-08:40(00:00)rootpts/1Apr2108:29-08:29(00:00)rootpts/1Apr2108:19-08:19(00:00)rootpts/110.0.1,199Apr2108:09-08:09(00:00)rootpts/1Apr2107:59-07:59(00:00)rootpts/1Apr2107:49-07:49(00:00)rootpts/1Apr2107:38-07:39(00:00)rootpts/1Apr2107:28-07:28(00:00)rootpts/1Apr2107:18-07:18(00:00)rootpts/1Apr2107:08-07:08(00:00)rootpts/1Apr2106:58-06:58(00:00)rootpts/1Apr2106:48-06:48(00:00)rootpts/1Apr2106:37-06:38(00:00)rootpts/1Apr2106:27-06:27(00:00)rootpts/1Apr2106:17-06:17(00:00)rootpts/1Apr2106:07-06:07(00:00)rootpts/1Apr2105:57-05:57(00:00)rootpts/1Apr2105:47-05:47(00:00)rootpts/1Apr2105:37-05:37(00:00)rootpts/1Apr2105:26-05:26(00:00)rootpts/1Apr2105:16-05:16(00:00)rootpts/1Apr2105:06-05:06(00:00)rootpts/1Apr2104:56-04:56(00:00)rootpts/1Apr2104:46-04:46(00:00)rootpts/1Apr2104:36-04:36(00:00)rootpts/1Apr2104:25-04:25(00:00)rootpts/1Apr2104:15-04:15(00:00)rootpts/1Apr2104:05-04:05(00:00)rootpts/1Apr2103:55-03:55(00:00)rootpts/1Apr2103:45-03:45(00:00)rootpts/1Apr2103:55-03:55(00:00)rootpts/1Apr2103:45-03:45(00:00)rootpts/1Apr2103:35-03:35(00:00)rootpts/1Apr2103:24-03:24(00:00)rootpts/1Apr2103:14-03:14(00:00)rootpts/1Apr2103:04-03:04(00:00)rootpts/1Apr2102:54-02:54(00:00)rootpts/110.0.1,199Apr2102:44-02:44(00:00)rootpts/1Apr2102:34-02:34(00:00)rootpts/1Apr2102:23-02:23(00:00)rootpts/1Apr2102:13-02:13(00:00)rootpts/1Apr2102:03-02:03(00:00)rootpts/1Apr2101:53-01:53(00:00)rootpts/1Apr2101:43-01:43(00:00)rootpts/1Apr2101:33-01:33(00:00)rootpts/1Apr2101:22-01:22(00:00)rootpts/1Apr2101:12-01:12(00:00)rootpts/1Apr2101:02-01:02(00:00)rootpts/1Apr2100:52-00:52(00:00)rootpts/110.0.1,199Apr2100:42-00:42(00:00)rootpts/1Apr2100:32-00:32(00:00)rootpts/1Apr2100:21-00:22(00:00)rootpts/1Apr2100:11-00:11(00:00)rootpts/1Apr2100:01-00:01(00:00)rootpts/1Apr2023:51-23:51(00:00)rootpts/1Apr2023:41-23:41(00:00)rootpts/1Apr2023:31-23:31(00:00)rootpts/1Apr2023:20-23:20(00:00)rootpts/1Apr2023:10-23:10(00:00)rootpts/1Apr2023:00-23:00(00:00)rootpts/1Apr2022:50-22:50(00:00)rootpts/1Apr2022:40-22:40(00:00)rootpts/1Apr2022:30-22:30(00:00)rootpts/1Apr2022:19-22:20(00:00)rootpts/1Apr2022:09-22:09(00:00)rootpts/1Apr2021:59-21:59(00:00)rootpts/1Apr2021:49-21:49(00:00)rootpts/1Apr2021:39-21:39(00:00)rootpts/1Apr2021:29-21:29(00:00)rootpts/1Apr2021:18-21:19(00:00)rootpts/1Apr2021:08-21:08(00:00)rootpts/1Apr2020:58-20:58(00:00)rootpts/1Apr2020:48-20:48(00:00)rootpts/1Apr2020:38-20:38(00:00)rootpts/1Apr2020:28-20:28(00:00)rootpts/1Apr2020:17-20:18(00:00)rootpts/1Apr2020:07-20:07(00:00)rootpts/1Apr2020:17-20:18(00:00)rootpts/1Apr2020:07-20:07(00:00)rootpts/1Apr2019:57-19:57(00:00)rootpts/1Apr2019:47-19:47(00:00)rootpts/1Apr2019:37-19:37(00:00)rootpts/1Apr2019:27-19:27(00:00)rootpts/110.0.1,199Apr2019:16-19:17(00:00)rootpts/1Apr2019:06-19:06(00:00)rootpts/1Apr2018:56-18:56(00:00)rootpts/1Apr2018:46-18:46(00:00)rootpts/1Apr2018:36-18:36(00:00)rootpts/1Apr2018:26-18:26(00:00)rootpts/1Apr2018:15-18:16(00:00)rootpts/1Apr2018:05-18:05(00:00)rootpts/1Apr2017:55-17:55(00:00)rootpts/1Apr2017:45-17:45(00:00)rootpts/1Apr2017:35-17:35(00:00)rootpts/1Apr2017:25-17:25(00:00)rootpts/110.0.1,199Apr2017:15-17:15(00:00)rootpts/1Apr2017:04-17:04(00:00)rootpts/1Apr2016:54-16:54(00:00)rootpts/1Apr2016:44-16:44(00:00)rootpts/1Ap