數(shù)據(jù)倉(cāng)庫(kù)系統(tǒng)設(shè)計(jì)說(shuō)明書

經(jīng)典word整理文檔，僅參考，雙擊此處可刪除頁(yè)眉頁(yè)腳。本資料屬于網(wǎng)絡(luò)整理，如有侵權(quán)，請(qǐng)聯(lián)系刪除，謝謝！歸一大數(shù)據(jù)平臺(tái)數(shù)據(jù)倉(cāng)庫(kù)系統(tǒng)設(shè)計(jì)說(shuō)明書系統(tǒng)設(shè)計(jì)說(shuō)明書2-系統(tǒng)設(shè)計(jì)說(shuō)明書1背景2..........................................................................................................................................................................................................34...........................................................................................1156783-系統(tǒng)設(shè)計(jì)說(shuō)明書4-系統(tǒng)設(shè)計(jì)說(shuō)明書01234567891引言5-系統(tǒng)設(shè)計(jì)說(shuō)明書6-系統(tǒng)設(shè)計(jì)說(shuō)明書2總體設(shè)計(jì)）））7-系統(tǒng)設(shè)計(jì)說(shuō)明書3系統(tǒng)接口設(shè)計(jì)無(wú)接口。4子系統(tǒng)/模塊設(shè)計(jì)4.1.1ODL(操作數(shù)據(jù)層設(shè)計(jì)4.1.1.1功能描述實(shí)現(xiàn)從外部系統(tǒng)導(dǎo)入數(shù)據(jù)到大數(shù)據(jù)平臺(tái),ODL層數(shù)據(jù)與數(shù)據(jù)源保持一致.同步方式包括:增量同步全量同步數(shù)據(jù)源類型包括:數(shù)據(jù)庫(kù)4.1.1.2業(yè)務(wù)流程流程圖:8-系統(tǒng)設(shè)計(jì)說(shuō)明書數(shù)據(jù)源系統(tǒng)數(shù)據(jù)庫(kù)遠(yuǎn)程調(diào)用讀取同步腳本數(shù)據(jù)庫(kù)同步是否流程說(shuō)明:1.調(diào)度系統(tǒng)定時(shí)啟動(dòng)數(shù)據(jù)同步作業(yè).2.作業(yè)讀取數(shù)據(jù)同步腳本,遠(yuǎn)程調(diào)用數(shù)據(jù)庫(kù)接口.3.數(shù)據(jù)庫(kù)接口返回遠(yuǎn)程調(diào)用結(jié)果.4.接收數(shù)據(jù)庫(kù)接口返回的數(shù)據(jù)集,裝載到ODL本地表.異常處理:調(diào)度系統(tǒng)捕獲異常并處理日志說(shuō)明:由調(diào)度系統(tǒng)負(fù)責(zé)記錄作業(yè)正常和異常日志.數(shù)據(jù)源說(shuō)明:1.通過(guò)視圖封裝需要同步的物理表,與業(yè)務(wù)進(jìn)行隔離.當(dāng)物理表發(fā)生變動(dòng),盡量通過(guò)修改視圖兼容.2.數(shù)據(jù)同步程序訪問(wèn)數(shù)據(jù)源視圖.數(shù)據(jù)粒度說(shuō)明：與數(shù)據(jù)源保持完全一致。4.1.1.3作業(yè)設(shè)計(jì)詳見數(shù)據(jù)同步列表.xlsx>>9-系統(tǒng)設(shè)計(jì)說(shuō)明書4.1.2事物層設(shè)計(jì)4.1.2.1功能描述BDL層對(duì)ODL數(shù)據(jù)進(jìn)行清洗、轉(zhuǎn)換，實(shí)現(xiàn)統(tǒng)一維度的數(shù)據(jù)整合。4.1.2.2業(yè)務(wù)流程流程圖:BDL數(shù)據(jù)流程圖讀取作業(yè)腳本數(shù)據(jù)清洗L階維度轉(zhuǎn)換、捕獲維表段結(jié)束流程說(shuō)明:1.調(diào)度系統(tǒng)定時(shí)啟動(dòng)數(shù)據(jù)同步作業(yè).2.作業(yè)讀取ODL表，進(jìn)行數(shù)據(jù)清洗，對(duì)不符合規(guī)則的數(shù)據(jù)進(jìn)行丟棄或轉(zhuǎn)換.3.將轉(zhuǎn)換后的結(jié)果數(shù)據(jù)裝載到BDL本地表.異常處理:調(diào)度系統(tǒng)捕獲異常并處理.日志說(shuō)明:由調(diào)度系統(tǒng)負(fù)責(zé)記錄作業(yè)正常和異常日志.10-系統(tǒng)設(shè)計(jì)說(shuō)明書數(shù)據(jù)粒度說(shuō)明：與ODL保持一致.4.1.2.3作業(yè)設(shè)計(jì)詳見數(shù)據(jù)倉(cāng)庫(kù)轉(zhuǎn)換文檔.xlsx>>4.1.3IDL(寬表層設(shè)計(jì)4.1.3.1功能描述在IDL層對(duì)BDL事實(shí)表按主題進(jìn)行合并,形成一次寫多次讀的寬表,降低系統(tǒng)整體開銷.并實(shí)際根據(jù)需要進(jìn)行用戶級(jí)粒度的聚合,進(jìn)一步提高查詢效率.4.1.3.2業(yè)務(wù)流程流程圖:分布式數(shù)據(jù)倉(cāng)庫(kù)讀取作業(yè)腳本按主題關(guān)聯(lián)事物級(jí)粒度L階用戶級(jí)粒度聚合段結(jié)束流程說(shuō)明:11-系統(tǒng)設(shè)計(jì)說(shuō)明書1.調(diào)度系統(tǒng)定時(shí)啟動(dòng)數(shù)據(jù)同步作業(yè).2.按主題讀取相關(guān)BDL表,進(jìn)行關(guān)聯(lián),盡量補(bǔ)齊維度字段,生成事物級(jí)寬表.3.按用戶級(jí)粒度聚合,生成用戶級(jí)寬表.異常處理:調(diào)度系統(tǒng)捕獲異常并處理日志說(shuō)明:由調(diào)度系統(tǒng)負(fù)責(zé)記錄作業(yè)正常和異常日志.數(shù)據(jù)粒度說(shuō)明:1.事物級(jí)粒度.2.用戶級(jí)粒度.4.1.3.3作業(yè)設(shè)計(jì)詳見數(shù)據(jù)倉(cāng)庫(kù)轉(zhuǎn)換文檔.xlsx>>4.1.4PDL(應(yīng)用層設(shè)計(jì)4.1.4.1功能描述PDL層按業(yè)務(wù)需求,IDL寬表匯總查詢生成各種結(jié)果表.4.1.4.2業(yè)務(wù)流程流程圖:12-系統(tǒng)設(shè)計(jì)說(shuō)明書調(diào)度系統(tǒng)開始各種粒度聚合讀取作業(yè)腳本維度條件篩選生成指標(biāo)數(shù)據(jù)段結(jié)束流程說(shuō)明:1.調(diào)度系統(tǒng)定時(shí)啟動(dòng)數(shù)據(jù)同步作業(yè).2.讀取IDL寬表,按業(yè)務(wù)需求進(jìn)行聚合,篩選,生成指標(biāo)數(shù)據(jù).3.結(jié)果寫入PDL層結(jié)果表.異常處理:調(diào)度系統(tǒng)捕獲異常并處理日志說(shuō)明:由調(diào)度系統(tǒng)負(fù)責(zé)記錄作業(yè)正常和異常日志.數(shù)據(jù)粒度說(shuō)明:1.按業(yè)務(wù)需求的各種粒度.4.1.4.3業(yè)務(wù)賬數(shù)據(jù)統(tǒng)計(jì)設(shè)計(jì)業(yè)務(wù)賬數(shù)據(jù)統(tǒng)計(jì)如下特點(diǎn):業(yè)務(wù)賬指標(biāo)由多個(gè)交易類型代碼組合,一個(gè)交易類型代碼對(duì)應(yīng)多個(gè)業(yè)務(wù)指標(biāo),例如:13-系統(tǒng)設(shè)計(jì)說(shuō)明書業(yè)務(wù)賬指標(biāo)A交易類型代碼{M00010,M00001,M00002,M00003,…}業(yè)務(wù)賬指標(biāo)B交易類型代碼{M00010,,M00012,M00013,…}業(yè)務(wù)賬指標(biāo)C交易類型代碼{M00000,,M00002,M00023,…}隨需求也業(yè)務(wù)變化,業(yè)務(wù)賬指標(biāo)及其交易類型代碼也會(huì)變化.每個(gè)業(yè)務(wù)賬指標(biāo)統(tǒng)計(jì)算法相同,只是其交易類型代碼組合不同.每個(gè)業(yè)務(wù)賬指標(biāo)可能會(huì)被多個(gè)報(bào)表調(diào)用，例如積分總金額”針對(duì)以上特性,運(yùn)算統(tǒng)一化技術(shù)的功能，減少系統(tǒng)維護(hù)成本。4.1.4.4作業(yè)設(shè)計(jì)詳見數(shù)據(jù)倉(cāng)庫(kù)轉(zhuǎn)換文檔.xlsx>>14-系統(tǒng)設(shè)計(jì)說(shuō)明書4.1.5PUB(維度庫(kù)設(shè)計(jì)4.1.5.1功能描述采用異步LOOUP方法,主動(dòng)捕獲非法維度數(shù)據(jù),經(jīng)過(guò)人工識(shí)別生成維度數(shù)據(jù).4.1.5.2業(yè)務(wù)流程流程圖:調(diào)度系統(tǒng)開始分布式數(shù)據(jù)倉(cāng)庫(kù)ODL表、維表讀取作業(yè)腳本捕獲異常維度是段結(jié)束流程說(shuō)明:1.調(diào)度系統(tǒng)定時(shí)啟動(dòng)數(shù)據(jù)同步作業(yè).2.異步讀取ODL表和維表,對(duì)比維度映射表查找是否有不存在的維度數(shù)據(jù).3.如果發(fā)現(xiàn)非法維度數(shù)據(jù),經(jīng)過(guò)人工識(shí)別,寫入維度映射表和維度表.異常處理:調(diào)度系統(tǒng)捕獲異常并處理日志說(shuō)明:由調(diào)度系統(tǒng)負(fù)責(zé)記錄作業(yè)正常和異常日志.15-系統(tǒng)設(shè)計(jì)說(shuō)明書4.1.5.3作業(yè)設(shè)計(jì)詳見數(shù)據(jù)倉(cāng)庫(kù)轉(zhuǎn)換文檔.xlsx>>4.1.6業(yè)務(wù)賬數(shù)據(jù)集市庫(kù)4.1.7數(shù)據(jù)導(dǎo)出設(shè)計(jì)4.1.7.1功能描述將PDL表和維度導(dǎo)出,同步給mysql庫(kù),供統(tǒng)計(jì)報(bào)表查詢使用.4.1.7.2業(yè)務(wù)流程數(shù)據(jù)導(dǎo)出流程圖門戶系統(tǒng)統(tǒng)計(jì)表、維表ETL階段流程說(shuō)明:1.調(diào)度系統(tǒng)定時(shí)啟動(dòng)數(shù)據(jù)同步作業(yè).16-系統(tǒng)設(shè)計(jì)說(shuō)明書2.讀取PDL表和維表,將數(shù)據(jù)裝載到MY