基于Amazon S3構(gòu)建數(shù)據(jù)平臺

1、基于Amazon S3構(gòu)建數(shù)據(jù)平臺技術(shù)創(chuàng)新 變革未來 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.Architecture數(shù)據(jù)平臺架構(gòu)三步走DataIn(gestion)DataDataStorage& ProcessingOut(put)&SchedulingPrinciplesPrinciplesInfrastructure asCodeLeast PrivilegeSoft DeleteConvention over ConfigurationIf you cant measure it,

2、you cant improve itInfrastructureasCodeCloudFormationvsTerraformPowered by AWSOnly in AWSOpensourcePlugin支持多種 InfraCloudFormationTerraformLeast PrivilegeLeast PrivilegeIAM精確到 Action, 避免使用 *, 例如 s3:*iam-hunter 方便定位生效的 IAM policy 文件SoftDeleteSoftDeleteDelete withmarkerCleanup laterConvention over Conf

3、igurationConventionoverConfigurationNaming mattersHive DatabaseLocation & Table Location & monitoringIf you cant measure it, you cant improve itIf you cant measure it, you cant improve itMetricsChartsPrometheus + GrafanaWhy S3?HDFS vsS3HDFS維護成本 ?幾乎無需維護可用性 ?可用性高冷數(shù)據(jù)存儲成本?Standard-IA/Glacier 存儲 冷數(shù)據(jù)S3如何設(shè)

4、計S3存儲beforewestartRequest Rateand Performance Considerations同一個 S3 bucket 盡量設(shè)計成 Key 前幾位不同Hive 中有Database 和Table 兩層NamespaceS3存儲設(shè)計數(shù)據(jù)分層例如 ODS, DW, DM 等每層一個 S3 Bucket路徑原則: s3:/bucket_name/db_name.db/table_name/Convention over Configuration方便分析 S3 Access Logdemocodeprivate標(biāo)記一些額外信息ACL:tags:versioning: 開啟

5、多版本lifecycle_rule: later cleanuplogging: 訪問日志每家公司都有一堆沒人用又沒人敢刪除的數(shù)據(jù)過期數(shù)據(jù)自動刪除IAGlacier30天轉(zhuǎn)移到60天轉(zhuǎn)移到90天自動刪除If you cant measure it, you cant improve itS3bucketmetric可以精確到 Database 或者Table支持各種 S3 操作統(tǒng)計S3bucketmetricS3Access LogS3 自動上傳 Access log到指定 bucket通過 SQL 分析“Stateless”DataPlatformoverS3EMR+autoscaling夜間自動擴容,加速計算白天自動縮容,節(jié)省計算資源autoscaling + Presto白天使用較多，增加節(jié) 點夜間減少節(jié)點，添加到EMR 集群異步執(zhí)行審計 +權(quán)限管理SQLBuffet數(shù)據(jù)質(zhì)量問題如何解決?查詢一個結(jié)果例如 count(*)驗證結(jié)果例如 兩個count 是否相等數(shù)據(jù)有問題?執(zhí)行其他查詢看數(shù)據(jù)明細如何定位數(shù)據(jù)問題data-ci開發(fā) data-ci系統(tǒng)給 ETL 寫單元測試自動執(zhí)行測試，失敗報警統(tǒng)計測試覆蓋率data-ciSQLasDataSQL解析中得到讀取了哪些 Table寫入了哪些 Tabl