阿里巴巴智能化運(yùn)維探索-李圣陶

李圣陶,花名劉昆l2009年加入阿里巴巴數(shù)據(jù)庫(kù)團(tuán)隊(duì)l阿里去I0E策略和雙十一的踐行者l2013年開始負(fù)責(zé)阿里集團(tuán)雙十一大促數(shù)據(jù)庫(kù)支持l2014年開始負(fù)責(zé)電商DBA團(tuán)隊(duì)支持阿里集團(tuán)泛電商業(yè)務(wù)聯(lián)系方式:taobaocomtaoliszgmailcomCoNTENTCoNTENTSAIisQL2016--新機(jī)遇新挑戰(zhàn)單機(jī)房AIisQL2016--新機(jī)遇新挑戰(zhàn)單機(jī)房單個(gè)應(yīng)用單機(jī)MysQL垂直拆分商業(yè)IOE異地多活云化自研POLARDB異地雙活單元化2011--201505--20102003--2004<50DBAs<50DBAs>100BUs>100BUsx1000MediaandEntertainmentOpenSource/Third-partyToolsandOpenSource/Third-partyDDBAdvisorIntelligentDiagnosticsandOptimizationDMSGUIforCentralizedManagementandDevOps,EfficientandSecureHDMHybridCloudBManagementDTSDataMigration&TransmissionADAMBMigrationEvaluationDBSBBackupServiceOLAPEnOLAPEngineAnalyticDBPB-gradeData.HighConcurrency.RedisGraphDBPOLARDBRedisGraphDBPOLARDBCloudNativeDB:DecoupledComputeandStoragewithHardwareAccelerationDataLakeAnalyticsServerlessInteractiveQueryServicewithPrestoandSparkIntegrationMongoDBHBase+X-PackMongoDB(Multi-modelAnalysis)TSDB(Multi-modelAnalysis)TSDBTimeSeries,SpatialTemporalDBMariaDB/MSSQLServerCloudDatabaseOperationCloudDatabaseOperationPlatformEnd-to-endTrackingandMonitoringServicePlatformDatabaseExpertServiceoracleoracleManagementlnfraManagementlnfraopenAlisoL/MysoLAlisoL/MysoLggtuningtuning–poLARDB.AnalyticDB.AlisoL.MysoL.NosoL.etc.es++sDDP=(Detection)+DescriptionDecisionMakingKeyDescriptionDecisionMakingKeyCapabilitiesNoAutomationHumanHumanAssistanceHumanStatisticsCollecting,Monitoring,Alerting,Scripts/ToolsSinglePointAutomationHuman+SDDPAnomalyDetection,SQLAdvisor,CapacityPlanning,HealthDiagnosticFramework,etc.PartialScenariosAutomationSDDP+HumanAutomaticRepairing,AutomaticSQLTuning,AutomaticConfigurationTuning,Auto-scaling,AutomaticResourceScheduling,AutomaticDataAccessProtection,etc.FullAutomationSDDPFullyEnd-to-EndAutomationforAllScenarios,NoHumanInterventionLevelLevel0Level1Level2Level3Level4…workloadworkloadL3NL2soLesOLterTuneslowslowsOLThrottlingfferfferfferfferfferBufferfferfferfferfferffer…odeIeIetricResourcescheduIingsDDPs0L odeIeIetricResourcescheduIingsDDPs0L AnomaIydetectioncoIIectionsoretricsicsfferfferfferfferfferfferTMallDingdingTaobaofferfferfferfferfferfferTMallDingdingTaobao?iBTune:morethan10,000instances,orysavingofTBdifferent.toavoidrestartingsystemmultipletimesrequirementofsLABufferpool(BP)sizeissensitivetomissratio:BPsizeisreducedfrom188Gto80Gwhenit’shitratioisfrom99.968%to99.950%CPUCPUusagelntuition:issrtioissrtio<<wdistributionfitsinourformulation.HrbstnbiehaorsNorm.oPs–FTRTbasedonthetolerablemissratioTraining–Input:twoinstances’metrics+rightinstance’sRT–Output:Leftinstance’sRT?Predicting–input:Ainstance’smetricsX2exceptmissratio–Output:RTcorrespondingtotoleratemissratio?ThegranularityofmetricisadaymissratiomissratioictedRTiscomparedwiththesafeSLARTmissmissratiostancesyictedRTiscomparedwiththesafeSLART(RT)(RT)DeterminethesafeRTfordifferentapplicationsGroupalltheinstancesintodifferentapplications,andfindthe95%percentileoftheresponsetimesineachgroupasthecorrespondingsafelimitforthatapplication.tedtedlyaggregationandnormalization;RTprediction;CalculateSLA(RT);CalculatenewBPsizeCalculatetolerablemissratio;Calculatesimilarity1515ionatAlibabasBPsizesBPsizethebackupnoderecovers2tingsystemservice.4ringthefollowinghoursbackupnodes’4ringthefollowinghoursbackupnodes’BPwillbeadjusted.increases,rollbackwillbetrigged.backupnodessincethebackupnodesBPisstilltheerollbackrestorestheoriginalstatusAllresultsarefromour?X-Engine:AliSQL:MySQLcompatibledatabasebasedonLSM-Treestorageengine?WithhighperformancePaxosimplementationsamplesacrossdifferentapplicationsPairwiseDNNKdatasamplesacrossdifferentapplicationsSingleinstanceadjustmentGGduringholidays?RedlineisthetimewhenBP?GreenlinesshowtheholidayisdaysredictedRTonlypointscceptableTheIOreadmetricistherealIO,allourDBinstancesturnongghmemorySDDPhashmemory–Cachepreload?BackupnodeneedstorunSQLstoloaddataintocacheafterBPadjustment?Performswitchingafterpreload–Bufferincrease?CurrentlyreplyonDBA?Automaticincreasebuffer–Multipleparameterstuning?DBMSconfigurefileessageQueueocessingBlinkPerceptiongActionPlannerActionSchedulerAutomationActionExecutoressageQueueocessingBlinkPerceptiongActionPlannerActionSchedulerAutomationActionExecutorPerformanceSecurityScalabilityAbnormalAccessDetectionAutomaticReschedulingServerlessAutomaticSQLPerformanceSecurityScalabilityAbnormalAccessDetectionAutomaticReschedulingServerlessAutomaticSQLTuningAutomaticConfigTuningSQLInjectionDetectionAutomaticSpaceTuningAuto-scalingAvailabilityAutomaticRepairingAutomaticSQLThrottlingelfperceptionSelfdeterminationSelfrepairinglftuningelfsecuringwledgewledgeBaseputsurityProtectingurityProtectingsourceSchedulingAutomaticRepairingpRestoreAutomaticTuningGlobalDecisionCoordinatorAut