基于 Flink SQL + Paimon 構建流式湖倉新方案

Contents01數(shù)據(jù)分析架構演進02介紹ApachePaimon0101DataLake(OSS/S3)DataLake(OSS/S3)Hive到LakehouseIcebergIceberg時效性：給業(yè)務帶來實際的加速價值APSARA云棲大會APSARA云棲大會ApacheIcebergDeltaLake面向Append數(shù)據(jù)處理，使用MergeInto更ApacheHudi提供增量更新的能力，但依然是全增量合并的方ApachePaimon原名FlinkTableStore，從Flink中孵化出來面向流而設計的數(shù)據(jù)湖，支持大規(guī)模更新，支持流讀APSARA云棲大會+XXX增量PositionDeleteFileEqualityDeleteFile全量DataFile(Parquet/ORC)寫放大嚴重LSM(RocksDB,Clickhouse,Doris,StarRocks)介紹介紹ApachePaimon02APSARA云棲大會NewFutureonCloud√√√√APSARA云棲大會PaimonContributor1400+Star570+FlinkTableStore誕生FlinkTableStore0.3發(fā)布進入Apache孵化器發(fā)布Paimon0.5版本第一個生產可用版本部分更新成熟Append場景可用中原銀行構建流式數(shù)倉2022.12023.12023.32023.9CompactionCompactionHudiCompactionCompactionHudiAPSARA云棲大會3000225075004X4.2XBucketBucketPaimon?Paimon9006003000近實時數(shù)倉，讀寫性能大幅提,FlinkDataandIntelligenceCapabilitiesofBusinessApplicationsOnlinePresenceofCoreTechnologies用戶用戶2Paimon是數(shù)據(jù)湖與ConvergenceofITConvergenceofITInfrastructure03APSARA云棲大會TagT-1TimeTravel-映射為分區(qū)表NewSnapshotGenerating…NewSnapshotGenerating…ConsumerSnapshotExpirationStreamingReadFailoverCausedby:java.io.FileNotFoundExceptionPaimonConsumer0Changelog生成流式更新Retract(Jason,1)流讀數(shù)據(jù)Table(name,count,PRIMARYKEY(name))正確結果：2Sum(count)Changelog生成是流計算的根本使用FlinkState生成？成本高，多份數(shù)據(jù)不—致代價大無法實時PaimonLookupChangelogProducer流/批Flink版流/批發(fā)布StreamingLakehouseLogs流/批Database全鏈路實時化