大數(shù)據(jù)實時計算Flink-SQL架構介紹課件

大數(shù)據(jù)實時計算FlinkSQL架構介紹技術創(chuàng)新

變革未來大數(shù)據(jù)實時計算FlinkSQL架構介紹技術創(chuàng)新1目

錄1Background2Flink

SQL

基本概念3Flink

SQL

核心功能4Flink

SQL

優(yōu)化5阿里云流計算產品目錄1Background2FlinkSQL基本概念32BackgroundBackground3Alibaba

Blink阿里巴巴Blink團隊有

20+flinkcontributor，6名

committer，向社區(qū)貢獻了數(shù)百個Commit+=Apache

FlinkAlibaba’s

ImprovementsBlinkAlibaba

BlinkBlink

Runtime+Flink

SQL=AlibabaBlink阿里巴巴Blink團隊有20+4團隊工作主導制定

Flink

SQL

語義DynamicTable

2016-2017Retraction

2016-2017完善

Flink

SQL功能Agg,Join,Window

2017跑通全部

TPCHQuery

2018性能提升大量的查詢優(yōu)化

2017-2018資源配置自動化

2018貢獻社區(qū)貢獻社區(qū)部分貢獻社區(qū)團隊工作主導制定FlinkSQL語義完善Flink5FlinkSQLFlinkSQL6Why

SQL?DeclarativeOneQuery,One

ResultOptimized Understandable StableUnifyWhySQL?DeclarativeOneQuery,7SQL

不是為流設計的沒有Retraction批計算查詢返回一個結果并結束數(shù)據(jù)是有限的批處理流數(shù)據(jù)是無窮的流上的查詢不斷產生結果且不會結束有對歷史數(shù)據(jù)的修改(Retraction)流處理SQL不是為流設計的沒有Retraction批計算查詢返回8動態(tài)表（Dynamic

Table）動態(tài)表（Dynamic

Table）: 數(shù)據(jù)會隨著時間變化的表動態(tài)表（DynamicTable）動態(tài)表（Dynamic9動態(tài)表

+

連續(xù)查詢連續(xù)查詢（Continuous

Query）：持續(xù)運行的查詢StreamStream連續(xù)查詢StreamStream連續(xù)查詢連續(xù)查詢Stream動態(tài)表+連續(xù)查詢連續(xù)查詢（ContinuousQuer10流計算

Retraction流計算Retraction11流計算

Retraction流計算Retraction12世界上不需要所謂的

Stream

SQL標準的

ANSI

SQL

就可以用來定義流計算世界上不需要所謂的StreamSQL13Flink

SQL

核心功能DDL&

DMLUDF/UDTF/UDAFWindow

AggJoinGroup

AggOver

AggFlinkSQL核心功能DDL&DMLUDF/UDT14LoadingData--

定義數(shù)據(jù)源表CREATETABLEclicks

(VARCHAR,TIMESTAMP,VARCHARusercTimeurl)WITH

(type=

'kafka',topic=

'click_topic',…);SELECT*FROM

clicksusercTimeurlMary12:00:00./homeBob12:00:00./cartMary12:00:05./prod?id=1LoadingData--定義數(shù)據(jù)源表VARCHAR,15SavingData--

定義數(shù)據(jù)結果表CREATETABLElast_clicks

(usercTimeurlVARCHAR,TIMESTAMP,VARCHAR,PRIMARYKEY

(user))WITH

(type=

'mysql',…);INSERTINTOlast_clicksSELECT*FROMclicksSavingData--定義數(shù)據(jù)結果表usercT16MultiOutputCREATEVIEWtaobao_clicks

ASSELECT*FROMclicksWHEREurlLIKE

'%’INSERTINTOmysql_resultSELECT*FROM

taobao_clicksINSERTINTO

hbase_resultSELECT*FROM

taobao_clicksCREATETABLEmysql_clicks

(usercTimeurlVARCHAR,TIMESTAMP,VARCHAR,PRIMARYKEY

(user))WITH

(type=

'mysql',…);CREATETABLEhbase_clicks

(usercTimeurlVARCHAR,TIMESTAMP,VARCHAR,PRIMARYKEY

(user))WITH

(type=

'hbase',…);MultiOutputCREATEVIEWtaoba17Group

AggregateMary1Mary2resultusercntMary3Bob1SELECTuser,COUNT(url)ascntFROM

clicksGROUPBY

userclicksusercTimeurlMary12:00:00./homeBob12:00:00./cartMary12:00:05./prod?id=1Mary12:01:45./prod?id=7從歷史到現(xiàn)在每個用戶點擊的次數(shù)GroupAggregateMary1Mary2resul18Window

Aggregate每小時每個用戶點擊的次數(shù)resultuserendTcntMary13:00:003Bob13:00:001Bob14:00:001Liz14:00:002Bob13:01:00./prod?id=4Liz13:30:00./cartLiz13:59:00./homeSELECTuser,TUMBLE_END(cTime,INTERVAL'1'

HOURS)ASendT,COUNT(url)AS

cntFROM

clicksGROUP

BYuser,TUMBLE(cTime,INTERVAL'1'

HOURS)clicksusercTimeurlMary12:00:00./homeBob12:00:00./cartMary12:02:00./prod?id=2Mary12:55:00./homeWindowAggregate每小時每個用戶點擊的次數(shù)re19雙流

JOIN：支持

INNER,

LEFT,

RIGHT,

FULL,

SEMI,

ANTISELECTo.orderId,ductId,o.orderTime,s.shipTimFROMOrderJOIN

ShipmON

o.orderOrdersorderIdproductIdorderTime53010:17:0061010:17:0591011:02:00121011:24:11ShipmentsorderIdshipTime510:55:00610:20:00911:58:001211:44:00esAS

oentsAS

sId=

s.orderIdresultorderIdproductIdorderTimeshipTime53010:17:0010:55:0061010:17:0510:20:0091011:02:0011:58:00121011:24:1111:44:00雙流JOIN：支持INNER,LEFT,RIGHT,20維表

JOIN：支持

INNER,

LEFTCREATETABLEProducts

(productIdVARCHAR,productNameVARCHAR,price

DECIMAL,PRIMARYKEY(productId),PERIODFOR

SYSTEM_TIME)WITH(type=

'hbase'…);SELECTo.*,p.*FROMOrdersAS

oJOINProductsFORSYSTEM_TIMEASOFPROCTIME()ASpONductId=

ductId維表JOIN：支持INNER,LEFTCREATET21聊幾個優(yōu)化聊幾個優(yōu)化22awabwbDataBaseReduced

ThroughputWaitfor

ResponseadDataBaseabcbcdSend

RequestReceive

RequestWaitConcurrent

ProcessingIncreased

ThroughputSync.

IOAsync.

IO異步維表

JOINawabwbDataBaseReducedThroughp23異步維表

JOINCREATETABLEProducts(productIdVARCHAR,productNameVARCHAR,price

DECIMAL,PRIMARYKEY(productId),PERIODFOR

SYSTEM_TIME)WITH

(type=

'hbase',async=

'true'…);SELECTo.*,p.*FROMOrdersAS

oJOINProductsFORSYSTEM_TIMEASOF

PROCTIME()AS

pONductId=

ductId一行配置的改動異步維表JOINCREATETABLEProducts24AggAggMapMapMap如何處理數(shù)據(jù)傾斜Data-SkewAggAggMapMapMap如何處理數(shù)據(jù)傾斜Data-Sk25如何處理數(shù)據(jù)傾斜Data-SkewAggAggMapMapMapHot!!反壓反壓反壓如何處理數(shù)據(jù)傾斜Data-SkewAggAggMapMapM26如何處理數(shù)據(jù)傾斜Data-SkewLocal-Global

Aggregation

優(yōu)化如何處理數(shù)據(jù)傾斜Data-SkewLocal-Global27如何處理數(shù)據(jù)傾斜Data-SkewLocalAggLocalAggLocalAggGlobalAggGlo