Apache Beam編程框架介紹

1、Apache Beam編程框架介紹流處理與批處理的統(tǒng)一編程框架Based on slides by Tyler Akidau & Frances Perry.Apache Beam定義了流處理與批處理統(tǒng)一的編程模型， 它生成的流水線具有很好的可移植性， 它在執(zhí)行層面定義了高效的執(zhí)行協(xié)議。1.Apache Beam 編程模型What Where When How2.SDK 構(gòu)建流水線JavaPython3. 執(zhí)行引擎 RunnersApexSparkFlinkGoogle Cloud DataflowBeam Model: 執(zhí)行用戶方法Apache FlinkApacheSparkBeam Mo

2、del: 構(gòu)建流水線OtherLanguagesBeam JavaBeam PythonExecutionExecutionCloudDataflowExecutionApache Apex流處理與批處理統(tǒng)一編程模型1目標(biāo):計(jì)算每個(gè)隊(duì)的得分支持流處理與批處理/wiki/File:Globe_centered_in_the_Atlantic_Ocean_(green_and_grey_globe_scheme).svg.越來越多的數(shù)據(jù).Thursday Wednesday Tuesday 無邊界的流數(shù)據(jù).9:008:0014:0013:0012:0011:0010:002:001:007:006

3、:005:004:003:00無序、有延遲的流數(shù)據(jù).9:008:0014:0013:0012:0011:0010:008:008:008:00事件時(shí)間 (Event time) 與 處理時(shí)間 (Processing Time)What Where WhenWhat 業(yè)務(wù)邏輯是什么? (ParDo, Combine)Where 在事件時(shí)間軸上對(duì)應(yīng)哪里? (Windowing)When 何時(shí)輸出結(jié)果？(Triggering)How 如何更新結(jié)果？(Accumulating)What: 求和 (Sum)What Where When/ Collection of raw log linesPColl

4、ection raw = IO.read(.);/ Element-wise transformation into team/score pairsPCollectionKV input =raw.apply(ParDo.of(new ParseFn();/ Composite transformation containing an aggregationPCollectionKV scores =input.apply(SegersPerKey();*All code snippets are pseudo-java - details shortened or elided for c

5、larity.What: 求和 (Sum)What Where WhenWhere: 在事件時(shí)間軸上對(duì)應(yīng)哪里?154Windowing 可以將流數(shù)據(jù)劃分為有邊界的數(shù)據(jù)集FixedSlidingSessions22 3134Key 1Key 2Key 3Time234What Where WhenWhere: 兩分鐘 固定窗口PCollectionKV scores = input.apply(Wo(FixedWindows.of(Duration.standardMinutes(2).apply(SegersPerKey();What Where WhenWhere: 兩分鐘

6、 固定窗口What Where When流數(shù)據(jù)處理: 引入Watermark的概念Event TimeProcessingTime11:0010:0015:0014:0013:0012:0011:0010:0015:0014:0013:0012:00InputOutputWhen: 何時(shí)輸出計(jì)算結(jié)果?Triggers控制結(jié)果輸出triggering(AtWatermark()What Where WhenWhen: 按時(shí)輸出（Triggering at theWatermark）PCollectionKV scores = input.apply(Wo(FixedWindow

7、s.of(Duration.standardMinutes(2).triggering(AtWatermark().apply(SegersPerKey();What Where WhenWhen: 按時(shí)輸出（Triggering at the Watermark）What Where WhenWhen: 提早輸出(Early Firing) 和 遲到輸出(LateFiring)PCollectionKV scores = input.apply(Wo(FixedWindows.of(Duration.standardMinutes(2).triggering(AtWater

8、mark().withEarlyFirings(AtPeriod(Duration.standardMinutes(1).withLateFirings(AtCount(1).apply(SegersPerKey();What Where WhenWhen: 提早輸出(Early Firing) 和 遲到輸出(Late Firing)What Where WhenHow: 如何更新結(jié)果？What Where When同一個(gè)窗口的多次輸出是如何累計(jì)的？取決于下游消費(fèi)者FiringElementsDiscardingAccumulatingAcc. & Retracting提前輸出3333按時(shí)輸出

9、5, 169(9, -3)遲到輸出2211(11, -9)Last Observed21111Total Observed112311(Accumulating & Retracting not yet implemented.)How: 加上新的，減去舊的scores = inputPCollectionKV.apply(Wo(Sessions.withGapDuration(Duration.standardMinutes(1).triggering(AtWatermark().withEarlyFirings(AtPeriod(Duration.standardMinu

10、tes(1).withLateFirings(AtCount(1).accumulatingAndRetractingFiredPanes().apply(SegersPerKey();What Where WhenHow: 加上新的，減去舊的What Where When1.傳統(tǒng)批處理2. 支持Window 的批處理3. 流數(shù)據(jù)處理4. 支持Retractions的流處理What Where WhenWhat When Where HowApache Beam流水線的可移植性21 + 1 = 2完整性實(shí)時(shí)性$運(yùn)維成本數(shù)據(jù)流水線開發(fā)的權(quán)衡Apache Beam流水線的可移植性1.編寫選擇最熟悉的語言*2.執(zhí)行選擇最適合的引擎統(tǒng)一的語義便于引擎的切換Beam Model: 執(zhí)行用戶方法ApacheFlinkApache SparkBeam Model: 構(gòu)建流水線Other LanguagesBeam JavaBeam PythonExecutionExecutionCloud DataflowExecutionApacheApex執(zhí)行引擎對(duì)Beam功能的支持表/learn/run