基于Kafka的跨數(shù)據(jù)中心復制平臺架構(gòu)

1、How Uber Builds A Cross Data Center Replication Platform on Apache Kafka基于Kafka的跨數(shù)據(jù)中心復制平臺架構(gòu)Apache Kafka at UberApache Kafka pipeline & replicationuReplicatorData loss detectionQ&AAgendaReal-time Dynamic PricingStream ProcessingDynamic pricingApp ViewsVehicle InformationApache KafkaUber Eats - Real-T

2、ime ETAsA bunch more .Fraud DetectionDriver & Rider Sign-ups, etc.Apache Kafka - Use CasesGeneral pub-sub, messaging queueStream processingAthenaX - self-service streaming analytics platform (Apache Samza & Apache Flink)Database changelog transportCassandra, MySQL, etc.IngestionHDFS, S3LoggingData I

3、nfrastructure UberPRODUCERSCONSUMERSReal-time Analytics, Alerts, DashboardsSamza / FlinkApplications Data ScienceAnalytics ReportingKafkaVertica / HiveRider AppDriver AppAPI / ServicesEtc.Ad-hoc ExplorationELKDebuggingHadoopMobile AppCassandra MySQLDATABASES(Internal) ServicesAWS S3PaymentSurgeTrill

4、ionsPBsMessages / DayDataTens of ThousandsTopicsScaleexcluding replicationApache Kafka at UberApache Kafka pipeline & replicationuReplicatorData loss detectionQ&AAgendaApache Kafka Pipeline UberDC2DC1Applications ProxyClientKafka REST ProxyRegional KafkaApplications ProxyClientKafka REST ProxyRegion

5、al KafkaSecondaryAggregate KafkauReplicatorOffset Sync ServiceAggregate KafkauReplicatorAggregationAggregate KafkauReplicatorOffset Sync ServiceAggregate KafkauReplicatorDC1Regional KafkaDC2Regional KafkaGlobal viewCross-Data Center FailoverAggregate KafkauReplicatorOffset Sync ServiceAggregate Kafk

6、auReplicatorDC1Regional KafkaDC2Regional KafkaDuring runtimeuReplicator reports offset mapping to offset sync serviceOffset sync service is all-active and the offset info is replicated across data centersDuring failoverConsumers ask offset sync service for offsets to resume consumption based on its

7、last commit offsetsOffset sync service translates offsets between aggregate clustersIngestionHDFS/S3DC1Aggregate KafkaHDFSS3DC2Aggregate KafkaHDFS/S3Topic MigrationConsumerDC1Regional KafkaDC2Regional KafkaSetup uReplicator from new cluster to old clusterMove producerMove consumerProducerTopic Migra

8、tionRegional KafkaConsumerDC1DC2Regional KafkaSetup uReplicator from new cluster to old clusterMove producerMove consumerRemove uReplicatorProduceruReplicatorTopic MigrationRegional KafkaConsumerDC1DC2Regional KafkaSetup uReplicator from new cluster to old clusterMove producerMove consumerRemove uRe

9、plicatorProduceruReplicatorConsumerTopic MigrationRegional KafkaDC1DC2Regional KafkaSetup uReplicator from new cluster to old clusterMove producerMove consumerRemove uReplicatorProduceruReplicatorConsumerTopic MigrationConsumerDC1Regional KafkaDC2Regional KafkaSetup uReplicator from new cluster to o

10、ld clusterMove producerMove consumerRemove uReplicatorProducerReplication - Use CasesAggregationGlobal viewAll-activeIngestionMigrationMove topics between clusters/DCsApache Kafka at UberApache Kafka pipeline & replicationuReplicatorData loss detectionQ&AAgendaMotivation - MirrorMakerPain pointExpen

11、sive rebalancingDifficulty adding topicsPossible data lossMetadata sync issuesRequirementsStable replicationSimple operationsHigh throughputNo data lossAuditingDesign - uReplicatorApache HelixuReplicator controllerStable replicationAssign topic partitions to each worker processHandle topic/worker ch

12、angesSimple operationsHandle adding/deleting topicsDesign - uReplicatoruReplicator workerApache Helix agentDynamic Simple ConsumerApache Kafka producerCommit after flushApache ZooKeeper Apache HelixControllerWorker ThreadApache HelixAgentFetcherManagertopic: testTopic partition:0FetcherThreadSimpleC

13、onsumerLeaderFind ThreadLinkedBlockingQueueApache Kafka ProducerRequirementsStable replicationSimple operationsHigh throughputNo data lossAuditingPerformance IssuesCatch-up time is too long (4 hours failover)Full-speed phase: 23 hoursLong tail phase: 56 hoursProblem: Full Speed PhaseDestination brok

14、ers are bound by CPUsSolution 1: Increase Batch SizeDestination brokers are bound by CPUsIncrease throughputproducer.batch.size: 64KB = 128KBproducer.linger.ms: 100 = 1000EffectBatch size increases: 1022KB = 5090KBCompress rate: 4062% = 2735% (compressed size)Solution 2: 1-1 Partition MappingRound r

15、obinTopic with N partitionsN2 connectionDoS-like trafficDeterministic partitionTopic with N partitionsN connectionReduce contentionMM 1MM 2MM 3p0p1p3MM 4SourceDestinationp2p0p1p2p3MM 1MM 2MM 3p0p1p3MM 4SourceDestinationp2p0p1p2p3Full Speed Phase ThroughputFirst hour: 27.2MB/s = 53.6MB/s per aggregat

16、e brokerBeforeAfterProblem 2: Long Tail PhaseDuring full speed phase, all workers are busySome workers catch up and become idle, but the others are still busySolution 1: Dynamic Workload BalanceDuring full speed phase, all workers are busyOriginal partition assignmentNumber of partitionsHeavy partit

17、ion on the same workerWorkload-based assignmentTotal workload when addedSource cluster bytes-in-rateRetrieved from Chaperone3Dynamic rebalance periodicallyExceeds 1.5 times the average workloadSolution 2: Lag-Feedback RebalanceDuring full speed phase, all workers are busyMonitors topic lagsBalance t

18、he workers based on lagsLarger lag = heavier workloadDedicated workers for lagging topicsDynamic Workload RebalancePeriodically adjust workload every 10 minutesMultiple lagging topics on the same worker spreads to multiple workersRequirementsStable replicationSimple operationsHigh throughputNo data

19、lossAuditingRequirementsStable replicationSimple operationsHigh throughputNo data lossAuditingMore IssuesScalabilityNumber of partitionsUp to 1 hour for a full rebalance during rolling restartOperationNumber of deploymentsNew clusterSanityLoopDouble routeDesign - Federated uReplicatorScalabilityMult

20、iple routeOperationDeployment groupOne managerDesign - Federated uReplicatoruReplicator ManagerDecide when to create new routeDecide how many workers in each routeAuto-scalingTotal workload of routeTotal lag in the routeTotal workload / expected workload on workerAutomatically add workers to routeDesign - Federated uReplicatoruReplicator Frontwhitelist/blacklist topics (topic, src, dst)Persist in DBSanity checksAssigns to uReplicator ManagerRequirementsStable replicationSimple operationsHigh throughputNo data lossAuditingApache Kafka at UberApache Kafka pi