Failure Data Collection and Analysis錯誤數(shù)據(jù)收集和分析

1、Failure Data Collection and AnalysisMotivation (1)My machine crashesSince 3/1/043 system crashes18 application errors96 application hangsWho cares?I do!People who share similar experiencesIn general, customer uproarMotivation (2)An Internet service has failuresWho cares?Internet service usersInterne

2、t service system administratorsAnyone affected by the ISs loss of revenueTotal: 61 user-visible failures in 12 months at Online ServiceMotivation (3)ROC/RADS needs real failure/attack information to drive benchmarksevaluate our prototypeshelp us select what we work attack Data Sources1000s of indivi

3、dual machinesCory/Soda Hall, BOINCLarge clusters at real Internet servicesInternet servicesDistributed applications on 100s of machinesPlanetLabIndividual MachinesData CollectionCollect minidumps that containThe Stop message/parameters/data Loaded drivers Processor context for processor that stopped

4、Process info/kernel context for process/thread stopped The Kernel-mode call stack for thread that stoppedFrequency of collectionsynchronized with application and system crashes on computers Analysis resultsWhat happened that is immediately responsible for the crashexact error codebrief description,

5、primarily for debuggingBucketing info, e.g.: driver fault Details for debugging, e.g. stack contentsUse Microsofts publicly available analysis toolsCaveat: significant variability in results between internal and public version of tool!How we collect minidumps (1)Corporate Error Reporting/resources/s

6、atech/cer/Manage error reports/msgs generated by WER and other programsConfigure clients to redirect reports to CER shared directorySample Statistics(25 nodes, 5 days)Sample Statistics(25 nodes, 5 days)Crashed ProgramVersionProblemBESConsole.exe3hungappCDCopier.exe1hungappCreateCD50.exe1hungappCreat

7、eCD50.exe1hungappexplorer.exe6.0.2800.1106shlwapi.dllfirefox.exehungappIAMAPP.EXE09hungappiexplore.exe6.0.2800.1106hungappiexplore.exe6.0.2800.1106mshtml.dllmatlab.exehungappmozilla.exe1.6.20040.11308ntdll.dllmsmsgs.exe009msmsgs.exeOUTLOOK.EXE10.0.4510.0hungappthunde1.exexpc3250.dllHow we collect mi

8、nidumps (2)BOINCFor SETIhome esque apps that pool resourcesProvides client API to send/receive data to/from BOINC serverWrite tools to read info in minidump directory and send to usSample Statistics (50 system crashes)Thread stuck in device driver12Page Fault in Nond Area10System Thread Exception No

9、t Handled6Unexpected Kernel Mode Trap6Kernel Mode Exception Not Handled5IRQL Not Less or Equal3Driver IRQL Not Less or Equal3NTFS File System2Bad Pool Caller2PFN List Corrupt1Sample Statistics (50 system crashes)watchdog.sys7ar5211.sys6ibmpmdrv.sys6ati3duag.dll5SYMEVENT.SYS3ipsecw2k.sys3memory_corru

10、ption 3ialmdev5.DLL2PSCRIPT4.DLL2ntoskrnl.exe2CLASSPNP.SYS2win32k.sys2SynTP.sys1TDI.SYS1ino_fltr.sys1ks.sys1drvnddm.sys1ntkrnlmp.exe1Pool_Corruption1Metrics (Windows & Linux)Availabilitysystem uptime, % time BOINC runningCPU(s)# processes, processor queue length, % non-idleMemoryavailable physical m

11、emory, free swap spaceDisk(s)free spaceNetwork(s)IP address, packets&bytes sent&received/sec, bandwidth to/from SETIhome server, first-hop bandwidth*, network coordinates*StaticCPU type, #, and benchmarks; total memory; OS typeQuestionsOther metrics?Frequency with which to measure them?What research

12、 questions can we answer with this data set?original goal: workload to evaluate our node discovery service evaluate effectiveness of network coordinatesevaluate potential to run more than just “embarrassingly parallel” apps on this type of infrastructure depending onmachines uptimenetwork connectivi

13、tyavailable disk spacedistributed analysis?security uses?Internet ServicesData characteristicsReal companiesMultitude of usersVoluminous data (several terabytes)Systems are complexTreat as black boxUse SLT algorithms for analysisMore data = better modelsAnalysis ResultsStudy event logsNot necessaril

14、y failuresCan derive models of good & bad behaviorModels with varying granularityUse different algorithmsVary boundary parametersFor more details see poster:“Towards a General Approach for Event Log Analysis”Distributed AppsPlanetLab“An open platform for developing, deploying, and accessing planetar

15、y-scale services”392 nodes at 164 sites around the worldPer-site system administrationApplications: OceanStore, PIERWhy?Platform for injecting faults and testing our algorithmsApplications on RADS-like environmentResearch platformMore accessibleUniversity-developed apps most likely to be tested on P

16、lanetLabApplications1) OceanStoreGlobal persistent data store.In the process of running prototype on PlanetLab Good source of failure data2) PIERDistributed query processorCurrently running on PlanetLab Good source of failure data + analysis engineWhat do we do with these apps?Instrument applications to collect any type of informationChoice of granulari