TiDB在HTAP數(shù)據(jù)庫的應(yīng)用

1、TiDB as an HTAP DatabaseTiDB在HTAP數(shù)據(jù)庫的應(yīng)用Why a new database?Brief HistoryStandalone RDBMSNoSQLMiddleware & ProxyNewSQL1970s20102015PresentMySQLPostgreSQL OracleDB2.Redis HBase Cassandra MongoDB.Google Spanner Google F1TiDBRDBMSNoSQLNewSQLNewSQL DatabaseHorizontal ScalabilityACID TransactionHigh Availa

2、bilitySQL at ScaleOLTP & OLAP ETL8am2pm6pm2amDatabaseERPCRMOLTPOLAPData WarehouseWhere is my data?Is the data out-of-date?Why two separate systemsHuge data sizeComplex query logicLatency VS ThroughputPoint query VS Full range scanTransaction & lsolation levelOLAP + OLTP = HTAPHybrid Transactional /

3、Analytical ProcessingACID TranscationReal-time analysisSQLHTAPHow do we build the new databaseWhat is TiDBScalability as the first class featureSQL is necessaryCompatible with MySQL, in most casesOLTP + OLAP = HTAP (Hybrid Transactional/Analytical Processing)24/7 availability, even in case of datace

4、nter outagesOpen source, of courseArchitectureTiKVTiKVTiKVTiKVRaftRaftTiDBTiDBTiDB.Placement Driver (PD)Control flow: Balance / FailoverMetadata / Timestamp requestStateless SQL LayerDistributed Storage LayerR.a.ftTiKV - OverviewRegion: a set of continuous key-value pairsData is organized/stored/rep

5、licated by RegionsHighly layeredTiKV Key Space start_key, end_key)(-, +)Sorted MapRPC (gRPC)TransactionMVCCRaftRocksDBNode BNode CNode ARaftRaftRaft256MBTiKV - Multi-RaftMultiple raft groups in the cluster, one group for each region.ClientStore 1Region 1Region 3Region 5Region 4Store 3Region 3Region

6、5Region 2Store 2Region 1Region 3Region 2Region 4Store 4Region 1Region 5Region 2Region 4RPCRPCRPCRPCTiKV node 1TiKV node 2TiKV node 3TiKV node 4Raft GroupTiKV - Horizontal ScaleRegion 1Region 3Region 1 Region 2Region 1*Region 2Region 2Region 3Region 3Node ANode BNode ENode CNode DAdd ReplicaThree ste

7、ps to move a leader replicaTransfer LeaderAdd ReplicaRemove ReplicaPD - OverviewMeta data managementLoad balance managementTiKVTiKVTiKVTiKV .TiKVClusterNode/Region InfoManagement CommandRoute InfoTiKV ClientPDPD - TiKV Cluster ManagmentRegion ARegion BNode 1Node 2PDScheduling StrategeCluster InfoAdm

8、inHeartBeatScheduling CommandRegion CConfigMovementTiDB - OverviewThe stateless SQL layerSQLASTLogical PlanOptimized Logical PlanCost ModelSelected Physical PlanTiKVTiKVTiKVTiDB SQL LayerStatisticsTiDB - Distributed SQLSELECT COUNT(c1) FROM t WHERE c1 10 AND c2 = shanghai;Partial AggregateCOUNT(c1)F

9、ilterc2 = “shanghai”Read Indexidx1: (10, +)Physical Plan on TiKV (index scan)Read Row Databy RowIDFinal AggregateSUM(COUNT(c1)DistSQL ScanCOUNTTiKVTiKVTiKVCOUNT(c1)COUNT(c1)TiDB - Cost Based OptimizerPredicate PushdownColumn PruningEager AggregateConvert Subquery to JoinStatistics frameworkCBO Frame

10、workIndex SelectionJoin Operator SelectionHash joinIndex lookup joinSort-merge joinStream Operators VS Hash OperatorsOLTP + OLAPTiDBTiDBTiDBTiDBTiDBOLTPQueryOLAPQueryTiKVTiKVTiKVTiKV.Job QueueWorkerSchedulerWorkerWorkerWorker PoolJobsJobsLow PriorityHigh PriorityHTAPEcosystem: Beyond TiDB and SQLSpa

11、rk on TiDBTiDBTiDBWorkerSpark DriverTiKV Cluster (Storage)TiKVTiKVTiKVApplicationSyncerData locationJobTiSparkDistSQL APITiKVTiDBTSO/Data locationWorkerWorkerSpark ClusterTiDB.TiDB Cluster.DistSQL APIPDPDPDPD ClusterMeta dataTiKVTiKVTiDBSpark on TiDBSpark ecosystemTiKV Connector is better than JDBC connectorIndex supportComplex Calculation PushdownCBOPick up ri