案例分析一次C4506交換機CPU利用率過高的處理過程

案例分析：一次C4506交換機CPU利用率過高處理過程剛處理過一次C4506交換機CPU利用率過高（1）經過交換機showproccpu命令輸出信息能夠看到：在5秒、1分鐘、5分鐘內Cat4kMgmtLoPri進程CPU占用率分別為91.69%92.18%92.23%顯然是這個進程消耗了大量CPU運行資源，從而造成交換機沒有新資源處理新請求。處理器運行兩個進程，Cat4kMgmtHiPri和Cat4kMgmtLoPri，這兩個進程同其余進程一樣要占用cpu時間。當Cat4k平臺上某項進程占用cpu超出了應分配cpu時間，Cat4kMgmtLoPri會接管這項進程，使其余進程能夠得到cpu時間。一樣當某個進程占用cpu時間沒有超出要求cpu利用率，Cat4kMgmtHiPri會接管這個進程。而Cat4kMgmtLoPri進程CPU占用率超出90%，我們判斷一定是某一個進程大大超出了CPU應分配時間，而Cat4kMgmtLoPri進程試圖接管這一進程，而造成了Cat4kMgmtLoPri進程CPU占用時間超出了90%。依照這一思緒只要我們找出這個進程，將其關閉或找出這個進程作用，就能夠處理這個問題并能夠找出CPU占用率過高原因。showproccpuCPUutilizationforfiveseconds:99%/0%;oneminute:99%;fiveminutes:99%PIDRuntime(ms)InvokeduSecs5Sec1Min5MinTTYProcess101300.00%0.00%0.00%0ChunkManager255221883250.00%0.00%0.00%0LoadMeter316013112210.00%0.00%0.00%0SpanTreeHelper40100.00%0.00%0.00%0DeferredEvents51039241473270540.00%0.06%0.05%0Checkheaps6420.00%0.00%0.00%0PoolManager70200.00%0.00%0.00%0Timers80200.00%0.00%0.00%0SerialBackgroun90100.00%0.00%0.00%0AAA_SERVER_DEADT100200.00%0.00%0.00%0AAAhigh-capacit110100.00%0.00%0.00%0PolicyManager1216182580.00%0.00%0.00%0IPCDynamicCach130100.00%0.00%0.00%0IPCZoneManager1464410913550.00%0.00%0.00%0IPCPeriodicTim1541210913630.00%0.00%0.00%0IPCDeferredPor160100.00%0.00%0.00%0IPCSeatManager170100.00%0.00%0.00%0IFSAgentManage181624845140053160.23%0.21%0.22%0ARPInput19681161810.00%0.00%0.00%0EntityMIBAPI200100.00%0.00%0.00%0SERIALA'detect2185610913670.00%0.00%0.00%0DynamicARPInsp22206027333750.00%0.00%0.00%0HCCounterTimer230100.00%0.00%0.00%0CriticalBkgnd24171691507180.00%0.00%0.00%0NetBackground250500.00%0.00%0.00%0Logger2671610911760.00%0.00%0.00%0TTYBackground272072111013180.00%0.00%0.00%0Per-SecondJobs28382921947196670.00%0.04%0.00%0Per-minuteJobs29285850888073603244.63%3.81%3.72%0Cat4kMgmtHiPri308437156467467994125091.69%92.18%92.23%0Cat4kMgmtLoPri312604935850.00%0.00%0.00%0GaliosReschedul320200.00%0.00%0.00%0IOSACLHelper330200.00%0.00%0.00%0NAMManager341522760000.00%0.00%0.00%0rftask3512428405853300.00%0.00%0.00%0NetInput3611620230225040.07%0.03%0.00%0Computeloadavg3708400.00%0.00%0.00%0BACKCHECK380100.00%0.00%0.00%0chkptmessageha390200.00%0.00%0.00%0cpf_process_msg_400100.00%0.00%0.00%0cpf_process_ipcQ4101400.00%0.00%0.00%0AggMgrProcess420100.00%0.00%0.00%0SFF84724301500.00%0.00%0.00%0Collectionproce440300.00%0.00%0.00%0CEFswitchingba450200.00%0.00%0.00%0AAADictionaryR460200.00%0.00%0.00%0AAAServer470100.00%0.00%0.00%0AAAACCTProc480100.00%0.00%0.00%0ACCTPeriodicPr4968882413997074920.71%0.62%0.62%0SpanningTree50439655767780.00%0.01%0.00%0DTPProtocol51401094630.00%0.00%0.00%0Ethchnl523744109502340.00%0.01%0.00%0UDLD5312934120.00%0.00%0.00%0DHCPSnooping54201825100.00%0.00%0.00%0Port-Security55252170049279115111.59%1.75%1.77%0IPInput560100.00%0.00%0.00%0ICMPeventhandl57919083791024240.07%0.14%0.11%0CDPProtocol5816221720.00%0.02%0.00%0Exec611482534580.00%0.00%0.00%0CEFbackgroundp620200.00%0.00%0.00%0XDRmcast630100.00%0.00%0.00%0IPCLCMessageH640100.00%0.00%0.00%0XDRRPPingBack65892580.00%0.00%0.00%0XDRRPbackgroun660100.00%0.00%0.00%0XDRRPTestBack670100.00%0.00%0.00%0IPIRDP68880641599285500.07%0.07%0.07%0CEF:IPv4proces691843651110.00%0.00%0.00%0ADJbackground70483671300.00%0.00%0.00%0L2MM711201447820.00%0.00%0.00%0MRD72106811946890.00%0.00%0.00%0IGMPSN730100.00%0.00%0.00%0IGMPSN-HA740100.00%0.00%0.00%0SocketTimers750200.00%0.00%0.00%0L2TRACESERVER7628816798170.00%0.00%0.00%0TCPTimer7740517840.00%0.00%0.00%0TCPProtocols78101256411790.00%0.00%0.00%0HTTPCORE790100.00%0.00%0.00%0CHKPTEXAMPLE800100.00%0.00%0.00%0CHKPTDevTest810200.00%0.00%0.00%0ATIP_UDP_TSK820100.00%0.00%0.00%0DHCPSnoopingHA830100.00%0.00%0.00%0ProbeInput840100.00%0.00%0.00%0RARPInput855380338081590.00%0.00%0.00%0DHCPDReceive86113220795440.00%0.00%0.00%0IPBackground8730018561610.00%0.00%0.00%0IPRIBUpdate880100.00%0.00%0.00%0COPS89161371110.00%0.00%0.00%0ClusterL290761094660.00%0.00%0.00%0ClusterRARP910200.00%0.00%0.00%0LOCALAAA920200.00%0.00%0.00%0AAACachedServe930200.00%0.00%0.00%0TPLUS940300.00%0.00%0.00%0RADIUSTESTCMD950200.00%0.00%0.00%0AAASENDSTOPEV9611256754707140.15%0.04%0.01%0PMCallback9738030391250.00%0.00%0.00%0VLANManager98891380.00%0.00%0.00%0DHCPDTimer994162500.00%0.00%0.00%0VTPTrapProcess1000200.00%0.00%0.00%0DHCPSecurityHe1010100.00%0.00%0.00%0DiagCard1/-11020100.00%0.00%0.00%0DiagCard2/-11030100.00%0.00%0.00%0DiagCard3/-11040100.00%0.00%0.00%0DiagCard4/-11054124768650.00%0.00%0.00%0SyslogTraps1060200.00%0.00%0.00%0VTPMIBEDITBUFF1070300.00%0.00%0.00%0SPANswitch1080200.00%0.00%0.00%0SNMPTimers1090200.00%0.00%0.00%0IPSNMP1100100.00%0.00%0.00%0PDUDISPATCHER1110100.00%0.00%0.00%0SNMPENGINE1120100.00%0.00%0.00%0SNMPConfCopyPro113804916320.00%0.00%0.00%0SNMPTraps1141344111625120.00%0.00%0.00%0NTP11532430976100.00%0.00%0.00%0DHCPDDatabase1163565465360.00%0.00%0.00%0Systempolling經過showplatformhealth命令深入更為詳細檢驗Catalyst4506交換機進程我們發(fā)覺K2CpuManReview進程有異常，以下：showplahea%CPU%CPURunTimeMaxPriorityAverage%CPUTotalTargetActualTargetActualFgBg5SecMinHourCPULj-poll1.000.0121441005000000:26GalChassisVp-review3.000.1710281005000002:50S2w-JobEventSchedule10.000.251091005000004:35Stub-JobEventSchedul10.002.36103410050022135:16StatValueManUpdate1.000.05101005000001:58Pim-review0.100.00101005000000:08Ebm-host-review1.000.31841005000000:27Ebm-port-review0.100.00101005000000:00Protocol-aging-revie0.200.00201005000000:00Acl-Flattener1.000.001051005000000:00KxAclPathMancreate/1.000.001051005000000:14KxAclPathManupdate2.000.0010141005000000:00KxAclPathManreprogr1.000.00211005000000:00TagMan-RecreateMtegR1.000.001051005000000:00K2CpuManReview30.0069.843026100500112106781248:14K2AccelPacketMan:Tx10.002.74200100500121210254:45K2AccelPacketMan:Au0.100.00001005000000:00K2AclMan-taggedFlatA1.000.001051005000000:00K2AclCamManstaleen1.000.001051005000000:00K2AclCamManhwstats3.000.501051005000007:24K2AclCamMankxstats1.000.011051005000003:03K2AclCamManAuditre1.007.761051005005004:07K2AclPolicerTableMan1.000.001011005000000:17K2L2AddressTableR2.003.251251005000007:25K2L2NewStaticAddr2.000.001051005000000:00K2L2NewMulticastA2.000.001051005000000:00K2L2DynamicAddress2.000.001051005000000:00K2L2VlanTableRevi2.000.001281005000000:02K2L2DestinationCa2.000.001001005000000:00K2PortManReview2.001.53151110050021126:42Gigaport65535Review0.400.04411005000001:15Gigaport65535Review0.400.07411005000001:15Gigaport65535Review0.400.06411005000001:15Gigaport65535Review0.400.05411005000001:17Gigaport65535Review0.400.05411005000001:17Gigaport65535Review0.400.06411005000001:16Gigaport65535Review0.400.05411005000001:17Gigaport65535Review0.400.05411005000001:17Gigaport65535Review0.400.12401005000005:23Gigaport65535Review0.400.09401005000005:20Gigaport65535Review0.400.08401005000005:17Gigaport65535Review0.400.09401005000005:19Gigaport65535Review0.400.04411005000001:18Gigaport65535Review0.400.08411005000001:17Gigaport65535Review0.400.06411005000001:16Gigaport65535Review0.400.06411005000001:16Gigaport65535Review0.400.05411005000001:24Gigaport65535Review0.400.05411005000001:22Gigaport65535Review0.400.07451005000001:22Gigaport65535Review0.400.07411005000001:22Gigaport65535Review0.400.074111005000001:22Gigaport65535Review0.400.06411005000001:23Gigaport65535Review0.400.05491005000001:22Gigaport65535Review0.400.05411005000001:21Gigaport65535Review0.400.08401005000005:19Gigaport65535Review0.400.11401005000005:22Gigaport65535Review0.400.08401005000005:19Gigaport65535Review0.400.10401005000005:19Gigaport65535Review0.400.08401005000005:18Gigaport65535Review0.400.09401005000005:18Gigaport65535Review0.400.11401005000005:20Gigaport65535Review0.400.09401005000005:19K2Fibcamusagerevi2.000.001501005000000:00K2FibIrmFibReview2.000.001501005000000:00K2FibVrfDefaultRo2.000.001501005000000:00K2FibAdjRepopRevie2.000.001501005000000:00K2FibVrfUnpuntRev2.000.001501005000000:08K2FibConsistencyCh1.0012.045210050021127:08K2FibAdjManStatsRe2.000.381081005000005:20K2FibAdjManHostMov2.000.001071005000000:14K2FibAdjManAdjChan2.000.001001005000000:00K2FibMulticastSigna2.000.031021005000000:46K2FibMulticastEntry2.000.001061005000000:00K2FibMulticastIrmM2.000.001071005000000:00K2FibFastDropManRev2.000.00701005000000:00K2FibPbrroutemapr2.000.412051005000004:02K2FibPbrflataclpr2.000.082011005000000:45K2FibPbrconsolidati2.000.011001005000000:12K2FibPerVlanPuntMan2.000.001521005000000:00K2FibFlowCacheflow2.000.001001005000000:07K2FibFlowCacheflow2.000.001001005000000:00K2FibFlowCacheadjr2.000.001001005000000:07K2FibFlowCacheflow2.000.001001005000000:02K2MetStatsManReview2.000.15521005000006:51K2FibMulticastMETS2.000.001001005000000:00K2QosDblManRateDBL2.000.09701005000001:48IrmFibThrottlerThro2.000.07731005000000:43K2VlanStatsManRevi2.000.9315410050011013:44K2PacketMemoryDia2.000.3115810050021016:06K2L2AgingTableRe2.000.072031005000002:42RkiosPortManPortRe2.005.07123510050044357:49RkiosModuleStateR4.000.024011005000000:30RkiosOnlineDiagRe4.000.014001005000000:26RkiosIpPbrIrmPortR2.000.011011005000000:40RkiosAclManReview3.000.053011005000000:55MatManReview0.500.00401005000000:00Slot2ILCManagerR3.000.001001005000000:00Slot2ILCS2wManRe3.000.001001005000000:00Slot3ILCManagerR3.000.001001005000000:00Slot3ILCS2wManRe3.000.001001005000000:00Slot4ILCManagerR3.000.001001005000000:00Slot4ILCS2wManRe3.000.001001005000000:00Slot5ILCManagerR3.000.001001005000000:00Slot5ILCS2wManRe3.000.001001005000000:00Slot6ILCManagerR3.000.001001005000000:00Slot6ILCS2wManRe3.000.001001005000000:00EthHoleLinecardMan(12.000.031001005000000:29EthHoleLinecardMan(22.000.181041005000003:55-------------%CPUTotals211.80111.31AllocationceilingCurrentallocation------------------------------------kbytes%inusekbytes%inuseLinecard1'sStore258.004%12.04100%Linecard2'sStore258.004%12.72100%Linecard3'sStore258.0060%155.24100%Linecard4'sStore258.0050%131.53100%Linecard5'sStore258.000%0.000%Linecard6'sStore258.000%0.000%TSMobjects------------------------------------RkiosSysPacketBuf250.000%0.560%PacketBufRaw20355.00100%20355.00100%PacketBufRawJumbo732.8125%183.20100%Packet1026.562%30.1799%PacketInfoItem390.620%0.190%VbufNodes240080.750%0.000%VbufNodes160055.750%12.190%VbufNodes40073.008%10.2661%VbufNodes6462.000%3.390%GalGbicEntrys11.340%0.000%PimPhyports875.0024%215.25100%PimPorts796.8731%247.03100%PimModules150.001%2.34100%PimSlots5.002%0.11100%PimChassis33.506%2.09100%EbmVlans2688.001%30.18100%EbmVlanGroupEntrys1920.000%4.80100%EbmPorts184.0030%55.70100%EbmPortHostEntrys384.000%0.000%EbmIeNodes536.000%4.18100%EbmPortVlanAclFeatur896.000%0.000%EbmSortedHostTableIt1.870%0.000%EbmSortedGroupTableI1.750%0.050%IrmVrfs6.851%0.10100%IrmFibAdjs768.001%16.1283%IrmPortEtherAddrEntr500.000%0.000%IrmFibEntries10240.000%39.7686%AclL4Op384.000%0.17100%AclL4OpTriplet256.000%0.15100%AclClassifier768.000%0.18100%AclFeature2512.680%0.42100%Acl384.000%0.10100%Ace1280.000%1.40100%AceActionDescStorage256.000%0.000%AclListNode256.000%0.06100%AceListNode25600.000%0.5561%AclClassifierActionL512.000%0.09100%AclLayerFeatureListN512.000%0.0650%AclClassifierListNod256.000%0.000%TableMapManNameToTa27.000%0.000%TableMapAllocator59.000%0.000%FlatAcl512.000%0.0933%FlatAce3840.000%3.9864%FlatAceActionListNod76800.000%1.5964%FlatAclL4OpSetStorag1024.000%0.000%FlatAclCacheNode1024.000%0.06100%QoSPolicers1672.000%0.000%KxAclPath1024.000%2.25100%KxAclPathListNode256.000%0.000%ConfigToHwAfMap418.780%0.000%HwToCfgAceMap192.000%0.000%CommandTables48.0021%10.26100%K2FibPbrFlatRouteMap515.850%0.000%K2FibPbrExpandedFlat2304.000%0.000%K2FibPbrFlatRouteMap320.000%0.000%K2FibVrfs38.591%0.59100%K2NetflowFPTEntri11.970%0.000%K2TxPacket384.000%0.090%K2TxPacketInfo256.000%0.170%EbmVlanHostEntrys2048.001%35.5092%MatEntrys7680.000%8.90100%MatEntryTableIterato1.120%0.030%RkiosAclManNamedGal43.750%0.000%RkiosAclVlanMaps272.000%0.000%RkiosAclVlanMapEnt1015.620%0.000%RkiosQoSPolicyMaps1315.930%0.000%RkiosQoSClassMaps896.000%0.02100%AclToIosFilterMapLis384.000%0.000%RkiosQoSPolicers380.000%0.000%RkiosTableMapGalios3.000%0.000%EventNodes84.000%0.758%EventNodes84.002%17.7113%KxAclTagPairNode2176.000%0.13100%KxAclMappingTableEnt96.020%0.840%KxAclMappingTableEnt64.010%0.000%KxAclTaggedFlatAcl1024.000%0.06100%RkisoIpPbrRouteMaps97.650%0.000%IrmFlows256.000%0.000%------------------------------------TSMtotals184649.7311%21299.0799%IOS軟件設計K2CpuManReview進程CPU占用率最高為30%而實際卻為69.84，大大超出了軟件當初設計得閥值，而其余進程都在軟件設計設定范圍之內，所以我們能夠初步判斷是K2CpuManReview進程出現(xiàn)了意外，致使CPU管理進程Cat4kMgmtLoPri試圖接管這一進程，而最終造成交換機CPU占用率過高，造成資源最終耗盡。從而影響了整個網絡運行穩(wěn)定性。經過Cisco提供文檔來看，K2CpuManReview進程主要處理“Hit”CPU數(shù)據(jù)包，Catalyist4506交換方式是CEF,并不是過程交換，在正常情況下除設備管理、監(jiān)測、路由計算外，數(shù)據(jù)包轉發(fā)主要由交換引擎來完成，并不需要CPU作過多處理。K2CpuManReview占用CPU資源過多，但各接口流量并沒有顯著增大，顯然不是正常負載過重造成，一定是很多異常數(shù)據(jù)包Hit了CPU,為了查清楚CPU受到異常數(shù)據(jù)包性質。我們經過命令：Monitorsession1soureinterfacecpuMonitorsession1desinterfacegi3/8?把CPU收到數(shù)據(jù)包鏡像到了gi3/8端口，然后在gi3/8端口接上天元龍馬IDS和裝有Sniffer和IRIS等數(shù)據(jù)分析軟件計算機。此時天元龍馬IDS非常精準監(jiān)測到大量SQL蠕蟲病毒攻擊和大量Synflood攻擊，（詳見天元龍馬企業(yè)IDS檢測匯報），此時我們已經能夠初步判定是網內SQL蠕蟲病毒暴發(fā)最終造成集團企業(yè)Catalyst4506交換機CPU資源耗盡。（因為流量過大，sniffer和IRIS等軟件死機）為了驗證我們判斷，我們把SQL蠕蟲病毒攻擊TCP1433端口加入到了訪問控制列表里面，在訪問控制列表應用到接口之后，CPU利用率立刻下降到15%左右，最終穩(wěn)定在13%--18%,K2CpuManReview進程CPU占用率也下降到10.4%，遠低于IOS軟件設定30%目標，這深入驗證了我們判斷。同時依照天元龍馬企業(yè)提供攻擊源地址，我們找到了其中一臺，并發(fā)覺了該計算機已中蠕蟲病毒，正在向外發(fā)送大量攻擊數(shù)據(jù)包。至此我們能夠斷定：是網內部分機器中SQL蠕蟲病毒，造成了這次交換機工作異常。應用訪問控制列表后運行狀態(tài)：showplatformhea%CPU%CPURunTimeMaxPriorityAverage%CPUTotalTargetActualTargetActualFgBg5SecMinHourCPULj-poll1.000.0121581005000000:51GalChassisVp-review3.000.1810281005000003:07S2w-JobEventSchedule10.000.211091005000005:04Stub-JobEventSchedul10.002.71103410050022138:53StatValueManUpdate1.000.06101005000002:11Pim-review0.100.00101005000000:09Ebm-host-review1.000.00841005000000:30Ebm-port-review0.100.00101005000000:00Protocol-aging-revie0.200.00201005000000:00Acl-Flattener1.000.001051005000000:00KxAclPathMancreate/1.000.001051005000000:14KxAclPathManupdate2.000.0010141005000000:02KxAclPathManreprogr1.000.00211005000000:00TagMan-RecreateMtegR1.000.001081005000000:00K2CpuManReview30.0010.3430271005001011541366:13K2AccelPacketMan:Tx10.003.42200100500346