ClickHouse在頭條技術(shù)架構(gòu)演化

1、技術(shù)創(chuàng)新，變革未來(lái)ClickHouse 在頭條技術(shù)架構(gòu)演化目錄ClickHouse 簡(jiǎn)介Bytedance 如何使用ClickHouse問(wèn)題與解決方案Q&ADeveloped by Yandex, and open source since 2016查詢性能優(yōu)越的分析型引擎主要特點(diǎn)（not new)Column oriented + vector executionLocal attached storage (not Hadoop ecosystem)Linear scalable & Reliable(shard + replication)SQL interfaceFastClickHo

2、use 簡(jiǎn)介Data Skipping- 分區(qū)以及分區(qū)剪枝數(shù)據(jù)局部有序（LSM-like engine, zone map)資源的垂直整合并發(fā) MPP+ SMP（plan level)- Tuned執(zhí)行層實(shí)現(xiàn) （multi-variant agg implementation, SIMD)C+ Template CodeClickHouse 簡(jiǎn)介 - 性能優(yōu)越的因素適用場(chǎng)景單表分析 或 colocate join casedistributed join 性能并不出色不足no transactionbatch data ingestweak update/delete supportweak

3、optimizer & query rewriteClickHouse 簡(jiǎn)介 - 適用場(chǎng)景與不足選擇ClickHouse的原因產(chǎn)品需求交互式分析能力 (in seconds)查詢模式多變以大寬表為主數(shù)據(jù)量大Open sourced MPP OLAP engine - (performance, feature, quality)Bytedance 如何使用ClickHouse幾千個(gè)節(jié)點(diǎn), 最大集群1200個(gè)節(jié)點(diǎn)數(shù)據(jù)總量 幾十PB日增數(shù)據(jù) 100TB查詢響應(yīng)時(shí)間(mostly) ms - 30s覆蓋下列用戶產(chǎn)品運(yùn)營(yíng)，分析師開(kāi)發(fā)人員少量廣告類(lèi)用戶openapiBytedance 如何使用Click

4、HouseBytedance 如何使用ClickHouse產(chǎn)品運(yùn)營(yíng)研發(fā)人員用戶行為分析平臺(tái)自研BI系統(tǒng)SQL 網(wǎng)關(guān)其他ClickHouse集群數(shù)據(jù)倉(cāng)庫(kù)（HIVE)KafkaRDBMS(MYSQL)數(shù)據(jù)ETL 服務(wù)Mini batch Data多種數(shù)據(jù)源 （離線 + 實(shí)時(shí) + )交互式分析數(shù)據(jù)處理鏈路對(duì)業(yè)務(wù)方透明滿足數(shù)據(jù)中臺(tái)對(duì)數(shù)據(jù)查詢需求。Bytedance 如何使用ClickHouse問(wèn)題與解決方案數(shù)據(jù)源-ClickHouse 服務(wù)化數(shù)據(jù)就 緒？ETL CH NodeCH NodeCH NodeHDFS消息中間件其他服務(wù)化與自動(dòng)化中的問(wèn)題HDFS 數(shù)據(jù)訪問(wèn)數(shù)據(jù)導(dǎo)入過(guò)程中Fail overCH數(shù)

5、據(jù)就緒速度(Part生成）增加 HDFS 數(shù)據(jù)訪問(wèn)能力（HDFS client porting from HAWQ)ETL服務(wù)維護(hù)外部事務(wù)保障數(shù)據(jù)一致性（fail over)INSERT INTO LOCAL TABLE數(shù)據(jù)構(gòu)建與查詢分離 (experimental feature)數(shù)據(jù)源-ClickHouse 服務(wù)化客戶端上報(bào)字段多變 (自定義參數(shù)）數(shù)據(jù)產(chǎn)品需要相對(duì)固定schemaEngine fix this Gap by Map type性能需求： 訪問(wèn)MAP 鍵值需要與訪問(wèn)POD 類(lèi)型的列速度一致 實(shí)現(xiàn)方式：LOB ？ Two-implicit column?OtherMap 數(shù)據(jù)類(lèi)型

6、 - 動(dòng)態(tài)Schema數(shù)據(jù)特征： # keys 總量可控， 局部有限局部（PART level)展平模型 (自描述）Map 數(shù)據(jù)類(lèi)型 - 動(dòng)態(tài)Schema1. MAP鍵訪問(wèn)自動(dòng)改寫(xiě)e.g.“select c_mapa from table” will be rewrote to “select c_map_a from table”2. MAP列訪問(wèn) (代價(jià)較大)e.g. select c_map from tableMerge階段優(yōu)化（無(wú)需重構(gòu)MAP column)收益：自動(dòng)化接入Table schema 簡(jiǎn)化極大簡(jiǎn)化數(shù)據(jù)構(gòu)建(ETL)邏輯Map 數(shù)據(jù)類(lèi)型 - 動(dòng)態(tài)Schema語(yǔ)法：Crea

7、te table t(c1 UInt64, c2 Map(String, UInt8) ENGINE=MergeTree.insert into t values(1, abc:1, bcd:2)Select c2abc from tMap 數(shù)據(jù)類(lèi)型 - 動(dòng)態(tài)Schema兩副本保障數(shù)據(jù)/服務(wù)ReplicatedMergeTree in ClickHouse （Issue)- Async Master-Master replication-High Volume Data & High Availability (zookeeper 使用問(wèn)題)ReplicatedMergeTree的問(wèn)題ZooK

8、eeper壓力大，znode太多400 Nodes 集群， 半年數(shù)據(jù)， 800萬(wàn)znodesReplicatedMergeTree use ZK to store:Table schema副本狀態(tài)（part info & log info)分片（shard) 狀態(tài)Catalog service + mini log service + coordinate serviceHigh Volume Data & High Availability (zookeeper 使用問(wèn)題)數(shù)據(jù)繼續(xù)增長(zhǎng)會(huì)導(dǎo)致ZK無(wú)法服務(wù)社區(qū)mini checksum in zk能緩解內(nèi)存使用，但不能解決問(wèn)題基于MergeTr

9、ee開(kāi)發(fā)HA 方案High Volume Data & High Availability (Zookeeper問(wèn)題)ZooKeeper只用作coordinateLog Sequence Number(LSN) 分配數(shù)據(jù)Block ID 分配元數(shù)據(jù)管理節(jié)點(diǎn)維護(hù)local log service (action log)Log 在分片內(nèi)部節(jié)點(diǎn)間通過(guò)Gossip協(xié)議交互數(shù)據(jù)信息（parts) 按需交互外部接口與社區(qū)兼容（例如：multi-master寫(xiě) 入）High Volume Data & High Availability - HaMergeTreeZK（LSN/ID alloc, meta

10、data)ReplicaReplicaPart/Log INFOHigh Volume Data & High Availability - HaMergeTreeZK（LSN/ID alloc, metadata)Replica AReplica BInsert1223A 獲取LSN和Block IDA Push log to active replica B2. B get its log lags from ZK and pull from AB redo the log and get Block from AHigh Volume Data & High Availability -

11、 HaMergeTreeZooKeeper壓力不會(huì)隨著數(shù)據(jù)量增長(zhǎng) 3M znodes in ZK保障數(shù)據(jù)&服務(wù)高可用String 類(lèi)型的濫用（from HIVE), 處理低效Why not LowCardinalityColumn？算子盡量在壓縮域上執(zhí)行（actionable compression)pure dictionary compressionpredication (equality family)group by (single/composite keys)String 類(lèi)型處理效率 - Global DictionaryString 類(lèi)型處理效率 - Global Dicti

12、onary1. Per replica字典(異步)構(gòu)建- why not cluster level/shard level?2. Support xxMergeTree onlyINSERTPART ROLLINMERGEAsync check & recodeWriteData & Dict (MergedBlockO utPutStream)1. 壓縮域執(zhí)行String 類(lèi)型處理效率 - Global DictionarySelectDecompress(optional)Execute actions on codeAnalyzeand RewriteRead compressed s

13、tring as code分布式表字典 （per shard, per replica)分布式表壓縮域執(zhí)行3. 性能提升約 20% 30%String 類(lèi)型處理效率 - Global DictionaryQuery：60天內(nèi)用戶轉(zhuǎn)化率/行為路徑，以及對(duì)應(yīng)每天轉(zhuǎn)化率內(nèi)存使用量大，OOM對(duì)服務(wù)穩(wěn)定性影響Aggregator無(wú)法感知底層數(shù)據(jù)特性特定場(chǎng)景內(nèi)存OOM - Step-ed AggregationAggregator 由執(zhí)行HINT控制HINT 感知數(shù)據(jù)分區(qū)/指標(biāo)語(yǔ)義Blocked Aggregator 按partition pipeline計(jì)算指標(biāo)。收益：內(nèi)存使用比默認(rèn)方式降低約五倍特定場(chǎng)

14、景內(nèi)存OOM - Step-ed AggregationArray類(lèi)型處理 - BloomFilter & BitMap indexArray類(lèi)型用來(lái)表示實(shí)驗(yàn)IDQuery：命中某些實(shí)驗(yàn)的用戶指標(biāo)單條記錄Array（實(shí)驗(yàn)） 幾百 or 上千Array col.binArray col in BlockArray hasAny1. 需要輔助信息減少 Array column materialize- Two scale BloomFilter (Part level, MRK range level)2. 減少Long Array column in Runtime Block- Transform hasAnyinto BitMap index OR-ingArray Column value+BitMap 集合has(array, valu