大數(shù)據(jù)即席查詢秒級響應(yīng)方案

1、大數(shù)據(jù)即席查詢秒級響應(yīng)方案技術(shù)創(chuàng)新，變革未來客戶關(guān)懷和CEM市場分析360C客戶洞察實時營銷與推薦客戶忠誠度維系客戶精細分群與個性化推薦客戶關(guān)懷與流程優(yōu)化預測與影響力分析網(wǎng)絡(luò)增效網(wǎng)絡(luò)性能管理與SQM策略保障 快速決策與根因分析定位網(wǎng)絡(luò)問題與規(guī)劃數(shù)據(jù)貨幣化 數(shù)據(jù)變現(xiàn) OTT開放競合M2M和位置分析Router +WDM(Backbone)DDDCloudOS/OpenStack(Local Resource) + Middlewareapps for OM apps for BizAPIapps forconsumerOSS suitsBig Data SuitsBSS suitsE2E ICT

2、 Resource Orchestration EngineE2E ICT Resource Orchestration EngineRaaSNaaSOpenStack PCRFCaaSRRURRUPartnersOM TeamBizCustomerConsumerCPEMxUMxUONTSDNSmarter SoftCom 業(yè)務(wù)和運營的智能融合1SDN實時大象流挖掘5小區(qū)擁塞動態(tài)控制2IPRAN流量仿真3 SON 網(wǎng)絡(luò)自動實時優(yōu)化 4快速故障關(guān)聯(lián)處理6潛在離網(wǎng)用戶維挽 7 一站式服務(wù)優(yōu)化8開放變現(xiàn)467OperationsBig Data Suits85Apps & ServicesClou

3、d OS/OpenStack (Local Resource, IaaS)SGSN MMEIMS HSSSDNcontrollerIT apps (SaaS)Telco apps SMS/IPTVMiddleware (PaaS)Cloud OS/OpenStack (Local Resource, IaaS)BRAS S/PGW GGSNRNCSDNSRC ControllerITappsPaaSFW DPI vCPESBC NAT1CloudEdg eCloud OS/OpenStackGSMLTEUMTS3CloudBBCloud OS/OpenStackADSLG.FastVDSL2C

4、loudDSL/OL TSmall CellSmall CellDD以太+OTN(Metro)DSDN數(shù)據(jù)現(xiàn)在和未來將深刻的改變運營商How to choose storage forcomplex big data requirements?NoSQL DatabaseKey-Value store: low latency, 5msCan not support multi-dimension queryMulti-dimensional problemPre-compute all aggregation combinationsComplexity: O(2n)Dimension 10

5、Too much spaceSlow loading speedShared nothing databaseParallel scan + distributed computeQuestionable scalability and fault-toleranceCluster size 100 data nodeNot suitable for big batch jobCan not integrate with Hadoop ecosystemSearch engineAll column indexedFast searchingSimple aggregationDesigned

6、 for search but not OLAPcomplex computation: TopN, join, multi-level aggregationNo SQL supportSQL on HadoopModern distributed architecture, scale well in computation.Pipeline based: Impala, Drill, Flink, BSP based: Hive, SparkSQLBUT, still using file format designed for batch jobFocus on scan onlyNo

7、 index support, not suitable for pointor small scan queriesArchitects choiceLoadingApp1App2App3ReplicationApp1App2App3Choice 1: CompromisingChoice 2: Replicating of data目錄:uCarbonData項目背景和適合的場景u 關(guān)鍵技術(shù)介紹u 性能和DEMO演示u Apache CarbonData社區(qū)和路標客戶需求：多維組合即席分析詳單過濾查詢按列掃描查詢開源生態(tài)集成當前大數(shù)據(jù)生態(tài)系統(tǒng)，沒有一種存儲方式同時滿足上面所有的需求！按列掃

8、描查詢（Full Scan）：沒有過濾條件，僅僅做匯 總計算等只查詢幾列信息典型的場景如：數(shù)據(jù)清洗處理日志分析典型場景1：按列掃描查詢C1C2C3C4C5C6C7R1R2R3R4R5R6R7R8R9R10.詳單過濾查詢（Small Scan）：按關(guān)鍵字快速過濾查詢 (類 似HBase)多組過濾條件組合，查詢 所有列要求查詢性能秒級響應(yīng)典型的場景如：運維查詢用戶行為分析典型場景2：詳單過濾查詢C1C2C3C4C5C6C7R1R2R3R4R5R6R7R8R9R10即席分析/Adhoc查詢：匯總計算多維度組合OLAP分析低時延即席查詢典型的場景如：Dash-Board報表Ad-hoc分析典型場景3：

9、多維組合即席分析C1C2C3C4C5C6C7R1R2R3R4R5R6R7R8R9R10R11詳單過濾查詢(Small scan)按列掃描查詢(Full scan)多維組合即席分析(OLAP analysis)CarbonData(一份數(shù)據(jù)滿足所有cases)Apache CarbonData實現(xiàn)一份數(shù)據(jù)同時滿足多種業(yè)務(wù)需求， 與Spark引擎 對接后形成一套分布式多維分析解決方案。為什么開始CarbonData項目?目錄:u CarbonData項目背景和適合的場景u 關(guān)鍵技術(shù)介紹u 性能和DEMO演示u Apache CarbonData社區(qū)和路標v分布式能力v快速查詢秒級響應(yīng)v高效數(shù)據(jù)存儲

10、方式v無縫與大數(shù)據(jù)生態(tài)集成開源是為了構(gòu)建生態(tài)，CarbonData是數(shù)據(jù)存儲層技術(shù)，要發(fā)揮價值， 需要與計算層、查詢層有效集成在一起，形成E2E生態(tài)發(fā)揮最大價值。CarbonData設(shè)計思路CarbonData獨特的價值特性v多種索引(MDK,MinMax,倒排)， 快速找到目標數(shù)據(jù)v字典編碼，減少計算開銷v支持數(shù)據(jù)更新IUD(開發(fā)中ing)v與大數(shù)據(jù)生態(tài)無縫集成，具有 HDFS分布式、可靠性等所有 優(yōu)點YearsQuartersMonthsTerritoryCountryQuantitySales2003QTR1JanEMEAGermany14211,4322003QTR1JanAPACCh

11、ina54154,7022003QTR1JanEMEASpain44344,6222003QTR1FebEMEADenmark54558,8712003QTR1FebEMEAItaly67556,1812003QTR1MarAPACIndia529,7492003QTR1MarEMEAUK57051,0182003QTR1MarJapanJapan56155,2452003QTR2AprAPACAustralia52550,3982003QTR2AprEMEAGermany14411,5321,1,1,1,1 : 142,114321,1,1,3,2 : 541,547021,1,1,1,3

12、: 443,446221,1,2,1,4 : 545,588711,1,2,1,5 : 675,561811,1,3,3,6 : 52,97491,1,3,1,7 : 570,510181,1,3,2,8 : 561,552451,2,4,3,9 : 525,503981,2,4,1,1 : 144,11532多維Key索引介紹數(shù)據(jù)即索引 (multi-dimensional keys)Blocklet Logical ViewSort (MDK Index)1,1,1,1,1 : 142,114321,1,1,1,3 : 443,446221,1,1,3,2 : 541,547021,1,2

13、,1,4 : 545,588711,1,2,1,5 : 675,561811,1,3,1,7 : 570,510181,1,3,2,8 : 561,552451,1,3,3,6 : 52,97491,2,4,1,1 : 144,115321,2,4,3,9 : 525,50398Sorted MDK IndexC1 C2 C3 C4 C5C6C7111111421143211113443446221113254154702112145455887111215675561811131757051018113285615524511336529749124111441153212439525503

14、98Encoding列式索引和排序高效數(shù)據(jù)壓縮(1/3)Blocklet Physical View142114324434462252550398111111111110 108103106221222429221311339333311147422111354111C1C2C3C4C5C6d rdrd rdrd rd rd rC71|1 :1|1 :1|1 :1|1 :1|1: 142:114321|2 :1|2 :1|2 :1|2 :1|9: 443:446221|3 :1|3 :1|3 :1|4 :2|3: 541:547021|4 :1|4 :2|4 :1|5 :3|2: 545:5

15、88711|5 :1|5 :2|5 :1|6 :4|4: 675:561811|6 :1|6 :3|6 :1|9 :5|5: 570:510181|7 :1|7 :3|7 :2|7 :6|8: 561:552451|8 :1|8 :3|8 :3|3 :7|6: 52:97491|9 :2|9 :4|9 :3|8 :8|7: 144:115321|10:2|10:4|10:3|10 :9|10 : 525:50398sort column within column chunk)Run Length Encoding & CompressionDim1 BlockDim2 BlockDim3 B

16、lockDim4 BlockDim5 Block8(7)9(10)Columnar Store541547021(1-10)1(1-8)1(1-3)1(1-2,4-1(1,9)545588712(9-10)2(4-5)6,9)2(3)675561813(6-8)2(7)3(2)570510184(9-10)3(3,8,10)4(4)561552455(5)5297496(8)14411532Column Level7(6)inverted Index倒排索引目錄:u CarbonData項目背景和適合的場景u 關(guān)鍵技術(shù)介紹u 性能和DEMO演示u Apache CarbonData社區(qū)和路標測

17、試環(huán)境DEMO EnvironmentData ModelNumber of Nodes1 master + 3workers#Columns300 (150 String, 150Double)# High Cardinality ColumnsColumns (10 Million)Columns (0.5 Million)vCPU40coresMemory384G# Medium Cardinality Columns4 Columns (0.4 Million)2 Columns (0.2 Million)11 Columns (0.1 Million)Data Size1.9TB#R

18、ecords1 billion rows * 300 columns#Row Size2KB性能構(gòu)造300萬行數(shù)據(jù)用同樣的SQL語句分別查詢CSV，Parquet，CarbonData數(shù)據(jù)：benchmark csvdf.filter($name = Allen and $gender = Male and $province = NB and$singler = false).count DEMO演示: CSV , Parquet,CarbonData目錄:u CarbonData項目背景和適合的場景u 關(guān)鍵技術(shù)介紹u 性能和DEMO演示u Apache CarbonData社區(qū)和路標Apache CarbonData社區(qū)已發(fā)布了社區(qū)穩(wěn)定版本 Apache CarbonData 0.1.0,0.1.1深度解讀Apache CarbonData：/cn/news/2016/07/huwei-CarbonData-data- second-resApache CarbonData源代碼地址: /apache/incubator- carbondata訂閱Dev Mailing，參與社區(qū)討論：dev如果有任何需求、建議、defects反