小米融合云架構(gòu)課件

小米融合云架構(gòu)技術(shù)創(chuàng)新，變革未來小米融合云架構(gòu)技術(shù)創(chuàng)新，變革未來1融合云介紹業(yè)務(wù)快速發(fā)展管理成本接入復(fù)雜隔離需求生態(tài)鏈需求公網(wǎng)訪問融合云動(dòng)機(jī)小米內(nèi)部業(yè)務(wù)生態(tài)鏈業(yè)務(wù)融合云介紹業(yè)務(wù)快速發(fā)展融合云動(dòng)機(jī)小米內(nèi)部業(yè)務(wù)生態(tài)鏈業(yè)務(wù)2融合云介紹服務(wù)化統(tǒng)—身份認(rèn)證權(quán)限管理多租戶Web

console基于開源功能擴(kuò)展基于HBase做二級(jí)索引系統(tǒng)擴(kuò)展基于HBase和HDFS做對(duì)象存儲(chǔ)公網(wǎng)調(diào)用支持Http請(qǐng)求/無狀態(tài)融合云設(shè)計(jì)融合云介紹服務(wù)化融合云設(shè)計(jì)3融合云介紹融合云架構(gòu)WebConsole SDK團(tuán)隊(duì)管理結(jié)構(gòu)化存儲(chǔ)對(duì)象存儲(chǔ)消息隊(duì)列流式消息離線 深度分析 學(xué)習(xí)自動(dòng)部署Docker鏡像…HBase/HDFS/Kudu/TensorFlow/Spark/Docker…融合云介紹融合云架構(gòu)WebConsole SDK團(tuán)隊(duì)管理消4團(tuán)隊(duì)管理Organization(Org)資源隸屬Team用戶身份隸屬于OrgAdmin/MemberOrgadmin

Team認(rèn)證CAS簽名認(rèn)證生成秘鑰秘鑰輪換權(quán)限管理<Team,Resource,

Perm>團(tuán)隊(duì)管理(Cloud-Manager)Cloud-ManagerServiceWebconsoleSDKCAS公司賬號(hào)user:

teamListresource:

org簽名簽名認(rèn)證Org(團(tuán)隊(duì)):

notebookAdmin

TeamDev_TeamTablesBucket/ObjectsqueuesReport

TeamAdminAdminR團(tuán)隊(duì)管理Organization(Org)團(tuán)隊(duì)管理(Clou5團(tuán)隊(duì)管理示例cc團(tuán)隊(duì)管理示例cc6??融合云存儲(chǔ)/計(jì)算數(shù)據(jù)流結(jié)構(gòu)化存儲(chǔ)(SDS)離線分析消息隊(duì)列(EMQ)對(duì)象存儲(chǔ)(FDS)消息隊(duì)列(EMQ)Snapshot

Export流式消息服務(wù)SDS

StreamMR/Spark??融合云存儲(chǔ)/計(jì)算數(shù)據(jù)流結(jié)構(gòu)化存儲(chǔ)離線分析消息隊(duì)列(EMQ7結(jié)構(gòu)化存儲(chǔ)(SDS)分布式數(shù)據(jù)庫基于HBase數(shù)據(jù)模型擴(kuò)展支持EntityGroup/Primary

Key支持?jǐn)?shù)據(jù)類型多語言SDK支持ACL擴(kuò)展功能擴(kuò)展局部二級(jí)索引全局二級(jí)索引Stream實(shí)時(shí)恢復(fù)StructureData

Store(SDS)結(jié)構(gòu)化存儲(chǔ)(SDS)分布式數(shù)據(jù)庫StructureData8結(jié)構(gòu)化存儲(chǔ)(SDS)SDS架構(gòu)ACL QuotaHBaseSDS

CoreSDSServiceObjectMappingFrontEndMulti-platform

SDKThriftProcessorHttpServer(TTransport)ThriftProcessorHttpClient(TTransport)flush/HttpRequestread/HttpResponseSDSService認(rèn)證與Cloud-Manager交互元信息管理UserTable=>TableId=>HBase

TableSDK– Java/php/python/go/c++/node.jsSDKService結(jié)構(gòu)化存儲(chǔ)(SDS)SDS架構(gòu)HBaseSDSCoreSD9結(jié)構(gòu)化存儲(chǔ)(SDS)SDSCoreSDSRequest<->HBaseRequestObjectMappingJavaObject<->SDSRequestACL表級(jí)別權(quán)限控制EntityGroup間權(quán)限隔離Quota<grantee,table,operation,

limit>TokenBucket算法SoftLimit引入集群過載quota集群過載前允許超發(fā)SDS架構(gòu)ACL QuotaHBaseSDS

CoreSDSServiceObjectMappingFrontEndMulti-platform

SDK結(jié)構(gòu)化存儲(chǔ)(SDS)SDSCore引入集群過載quotaS10結(jié)構(gòu)化存儲(chǔ)(SDS)實(shí)現(xiàn)Entity

Group(EG)功能業(yè)務(wù)特性:

以用戶為中心定義Entity

Group

KeysEG在同—個(gè)regionEG內(nèi)跨行寫原子性EG內(nèi)局部二級(jí)索引強(qiáng)—致Schema管理EntityGroup/PrimaryIndex/Secondary

Index/Column數(shù)據(jù)類型Bool/Int8/Int16/Float/Double/String/Binary,

Set使用OrderedBinary編碼確保順序條件查詢SDS

Core主要功能結(jié)構(gòu)化存儲(chǔ)(SDS)實(shí)現(xiàn)EntityGroup(EG)功能11結(jié)構(gòu)化存儲(chǔ)(SDS)局部二級(jí)索引數(shù)據(jù)Family/索引FamilyHBase

coprocessor中更新索引讀取主記錄構(gòu)造索引rowkey更新主記錄/刪除舊索引/寫入新索引支持Eager/Lazy/Immutable

IndexLazy

invalid

index定期清理Eager/Immutable

Index支持屬性投影全局二級(jí)索引使用單獨(dú)的HBase

table存儲(chǔ)索引使用Themis提供HBase跨表更新原子性/xiaomi/themis全局單調(diào)遞增時(shí)間戳:

/xiaomi/chronos強(qiáng)—致性二級(jí)索引EG Row01 main_row…01 Index_rowData

FamilyIndexFamily結(jié)構(gòu)化存儲(chǔ)(SDS)局部二級(jí)索引https://github12結(jié)構(gòu)化存儲(chǔ)(SDS)使用Javaannotation和反射實(shí)現(xiàn)toRecord:Object=>Map<String,

Object>fromRecord:Map<String,Object>=>

ObjectObject

Mapping對(duì)象映射結(jié)構(gòu)化存儲(chǔ)(SDS)使用Javaannotation和反射13SDS

Object

Mapping?例Table

Schema定義(SQL)SDSObjectMapping?例TableSche14SDS

Object

Mapping?例Table

Schema定義SDSObjectMapping?例TableSche15SDS

Object

Mapping?例查詢SDSObjectMapping?例查詢16SDS

Object

Mapping?例查詢SDSObjectMapping?例查詢17結(jié)構(gòu)化存儲(chǔ)(SDS)場(chǎng)景OLAP增量數(shù)據(jù)處理原理Stream

Family存儲(chǔ)image使用hbase

coprocessor

build

images–

推送image到流式消息服務(wù)實(shí)時(shí)恢復(fù)StreamF1F2MRSnapshoterData ImagesnapshotsRecoverySnapshot

justbefore

Ts流式消息服務(wù)Ts結(jié)構(gòu)化存儲(chǔ)(SDS)場(chǎng)景使用hbasecoprocesso18Web

console?例Webconsole?例19結(jié)構(gòu)化存儲(chǔ)(SDS)SDSrest

server–24Intel(R)Xeon(R)E5620@2.4GHz,64G,

萬兆網(wǎng)HBaseregionserver–

Cpu/內(nèi)存/網(wǎng)絡(luò)同上，8

*

800G

ssd，5臺(tái)?YCSB

client性能測(cè)試Put

Througput/Latency記錄大小ThrougputAvg(ms)P95(ms)P99(ms)0.1k337380.52790.5k300470.466111k256520.46610Get

Througput/Latency記錄大小ThrougputAvg(ms)P95(ms)P99(ms)0.1k302130.26470.5k274570.28391k269320.29312結(jié)構(gòu)化存儲(chǔ)(SDS)SDSrestserverYCSB20對(duì)象存儲(chǔ)(FDS)KB到TB級(jí)別的文件基于HBase

&

HDFSBucket&

ObjectRestful

API多語言SDK權(quán)限管理事件通知Quota管理30+業(yè)務(wù)使用FileData

Store(FDS)FrontEndMetaManagerBlobStoreHBaseHDFSCDN對(duì)象存儲(chǔ)(FDS)KB到TB級(jí)別的文件FileDataS21對(duì)象存儲(chǔ)(FDS)Bucket

TableBucket基本信息/ACL/QuotaObject

TableObject基本信息/ACL/meta/通知信息Objectname=>blobUriBlobInfo

TableBlobUri=>FileId抽象Blob

write/read/get接口，隱藏具體實(shí)現(xiàn)FileInfo

TableFileId=>Path,

basicInfoMultiPartUpload

TableuploadId=>

partInfo元數(shù)據(jù)管理對(duì)象存儲(chǔ)(FDS)BucketTable元數(shù)據(jù)管理22對(duì)象存儲(chǔ)(FDS)BlobManager/FileManagerBlob存儲(chǔ)magic/data/md5offset管理打開文件MaxOpenNum/MaxFileSizeIdleFiles<=>Busy

FilesBlobClient創(chuàng)建Blob:

BlobWriterWrapped

OutputStream關(guān)閉Blob獲取Blob:BlobReaderWrapped

InputStreamBlobStoreBlobManagerBlobInfo(BlobId->FileId)FileManagerFileInfo(FileId->FilePath)BusyFilesIdleFilesHDFS

FilesBlobClientcreateBlob/getBlobcloseBlob對(duì)象存儲(chǔ)(FDS)BlobManager/FileManag23對(duì)象存儲(chǔ)(FDS)創(chuàng)建對(duì)象BlobClient:

createBlob將object信息寫入hbase

object

table讀取對(duì)象從hbase

object

table獲取blobUriBlobClient:

getBlob對(duì)象列表t=苗hbase

object

table刪除對(duì)象刪除hbase

object

table中對(duì)應(yīng)記錄對(duì)象橾作對(duì)象存儲(chǔ)(FDS)創(chuàng)建對(duì)象對(duì)象橾作24對(duì)象存儲(chǔ)(FDS)Object

Cleaner–

民據(jù)objecttable刪除blobtableFileCleaner&

Compactor基于BlobInfo

Table和FileInfo

Table民據(jù)BlobInfo

Table統(tǒng)計(jì)File使用率歸并低使用率文件Archive無Blob引用文件刪除過期Archive文件Cleaner&

CompactorobjNameblobUriblobUriFileIdFileIdFileInfo對(duì)象存儲(chǔ)(FDS)ObjectCleaner–民據(jù)obj25FDS性能FDS性能26消息隊(duì)列(EMQ)應(yīng)用間異步通信支持P2P,

Pub/Sub兩種模式基于HBase提供分布式能力Atleastonce語義多讀者消息優(yōu)先級(jí)死信隊(duì)列Short/Long

PoolingElasticMessageQueue(EMQ)EMQsend

messagereceive

messageack

message消息隊(duì)列(EMQ)應(yīng)用間異步通信多讀者ElasticMes27消息隊(duì)列(EMQ)QueueHBase

TablePartitionsPartitionIdPartition分布—致性hashEMQ架構(gòu)Partition_0Partition_nRegion_0Region_nQueueHBase

Table……HBasePartition0Partition1Partition2Partition3Partition4Partition5EMQ

Server消息隊(duì)列(EMQ)QueueEMQ架構(gòu)Partition_028消息隊(duì)列(EMQ)Message–

MessageIdmessageId=hbaseRow=

<partitonId><timestamp><sequenceId>Timestamp=currentTime+del