HugeGraph圖數(shù)據(jù)庫平臺介紹

1、HugeGraph圖數(shù)據(jù)庫平臺介紹技術(shù)創(chuàng)新，變革未來目錄圖數(shù)據(jù)庫是什么HugeGraph是什么HugeGraph典型案例HugeGraph存儲原理如何參與開源貢獻3圖數(shù)據(jù)庫是什么新型的NoSQL數(shù)據(jù)庫4Key-ValueDocumentColumnGraphRedis RocksDBHBase CassandraCouchDB MongoDBNeo4j TitanDB查詢復(fù)雜度數(shù) 據(jù) 關(guān) 聯(lián) 度圖數(shù)據(jù)庫是什么圖數(shù)據(jù)庫是一種新型的NoSQL數(shù)據(jù)庫5圖數(shù)據(jù)庫是什么圖數(shù)據(jù)庫在近幾年廣受關(guān)注6圖數(shù)據(jù)庫是什么圖數(shù)據(jù)庫：存儲實體與實體之間的關(guān)聯(lián)關(guān)系G = (V, E)Vertex (Node, EnIty

2、, Object)Edge (RelaIonship, Link, Arc)現(xiàn)實世界圖數(shù)據(jù)庫7圖數(shù)據(jù)庫是什么8idnameagephone1Tom22188*11112Mike23158*22221. useriduser1user2since1122016-01-0222. friend好友關(guān)系用 用 戶戶圖數(shù)據(jù)庫帶來的改變SQL圖數(shù)據(jù)庫select * from friend ajoin friend b on b.user1=a.user2 join user on id=b.user2where a.user1 in (select id from user where name=To

3、m)Tomg.V().has(name, Tom).out(friend).out(friend)9圖數(shù)據(jù)庫帶來的改變10數(shù)據(jù)來自程序員深度關(guān)系型數(shù)據(jù)庫執(zhí)行時間 / s圖數(shù)據(jù)庫執(zhí)行時間 / s返回結(jié)果數(shù)量20.0160.012500330.270.16811000041543.511.35960000052.132800000圖數(shù)據(jù)庫應(yīng)用行業(yè)反欺詐發(fā)現(xiàn)人員、事件、地點 和時間之間的異常聯(lián)系知識圖譜構(gòu)建實體與實體間的關(guān) 聯(lián)關(guān)系，描述真實世界網(wǎng)絡(luò)安全服務(wù)器、域名、IP、文件 等關(guān)聯(lián)關(guān)系，形成網(wǎng)絡(luò)安 全情報IT運維系統(tǒng)調(diào)用，數(shù)據(jù)庫訪問， 緩存讀取等全鏈路監(jiān)控更多圖數(shù)據(jù)庫應(yīng)用金融社交招聘物流醫(yī)療電信零

4、售圖數(shù)據(jù)庫11HugeGraph是什么百度安全自研的開源圖數(shù)據(jù)庫12HugeGraph特點HugeGraph易用高效通用開放可擴展大規(guī)模1314PersonRobert ZemeckisPersonNora EphronHugeGraph優(yōu)勢MovieSleepless in SealeMovieYouve Got MaildirecteddirectedMovieWhen Harry Met SallywrotePersonMeg RyanactedPersonactedTom HanksactedMovieCast AwayactedactedMovieactedJoe Versus Vo

5、lcanoactedacteddirectedMovieThe Polar Express directed關(guān)系數(shù)據(jù)庫 MySQL 百萬千萬規(guī)模十億+規(guī)模圖數(shù)據(jù)庫 HugeGraph易 建 模大 規(guī) 模782345likelikefriendfriendfriendfriendlikelikelike?6like9like1personmoviev1v2v3v4v5v2v6v7v3v8v4v8v5v8v9高 效 關(guān) 聯(lián) 分 析/ Gremlin 2層好友查詢g.V().has(name, Tom Hanks).out(friend).out(friend)2362345friend1Tom H

6、anksfriendfriend4friend5 friendfriend71source6target靈 活 查 詢語 / 查詢2點之間的最短路徑g.V(source_id)言 .repeat(out().simplePath().until(hasId(target_id).path().limit(1)HugeGraph技術(shù)選型RocksDBCassandraHBase4. 持久化方案Non-Na3ve StorageNaIve Storage3. 查詢語言GremlinCypherSPARQLg.V().has(name,gremlin).out(knows).out(knows).v

7、alues(name)2. 存儲模型A: (B,D)B: (C,D)C: (E)D: (B,C,E)E: (A,C)F: (D,E)Adjacency ListAdjacency Matrix1. 概念模型Property GraphRDFpersonpersonpersonpersonsoftsoft15HugeGraph整體架構(gòu)OLAPMonitorStudioLoaderToolsApplicaIonsClientGraph EngineBackend Storage InterfaceJob Manager & ExecutorOLTPMonitor APIReskul APIGrem

8、lin API Internal API Layer (Schema, Graph, Index)Gremlin Job APIRocksDBCassandraHBasePalo | TableInMemoryScyllaDBMySQL(Ext Plugins)16HugeGraph性能批量寫入性能最短路徑查詢性能數(shù)據(jù)集后端email-enron (30w edge)amazon0601 (300w edge)com-youtube.ungraph (300w edge)com-lj.ungraph (3000w edge)HugeGraph1.72613.06613.009141.212Ti

9、tan14.02125.975154.9261467.159Neo4J4.69419.39622.199447.488數(shù)據(jù)集后端email-enron (30w edge)Amazon0601 (300w edge)com-youtube.ungraph (300w edge)com-lj.ungraph (3000w edge)HugeGraph2.2890.24210.21828.78Titan13.3260.577544.492848.36Neo4J2.0013.8995.93728.9251710億邊導(dǎo)入 BSKU, KGProduct-BSKU,KGUser, Doc - BSKU,

10、 KGKG-libquery-kgproduct-kgAD-kguser-kgdoc-kg結(jié)構(gòu)化解析實體鏈接 / 屬性填充Score / Rank知識沉淀 圖譜進化HugeGraph知識沉淀 / 圖譜進化實體挖掘 圖譜構(gòu)建實體識別 / 專名識別 屬性識別 / 屬性填充 實 體 挖 掘關(guān)系挖掘 / 共引消歧 實體融合 / 置信計算 圖 譜 構(gòu) 建數(shù)據(jù)處理數(shù)據(jù)引入 / 數(shù)據(jù)清洗數(shù)據(jù)解析 / 結(jié)構(gòu)提取實體和實體關(guān)系蘊含規(guī)則和知識典型圖數(shù)據(jù)應(yīng)用場景20基于知識圖譜的廣告推薦HugeGraph圖遍歷切詞切詞 組合查找 搜索詞追溯 行業(yè)路徑結(jié)果 打分 根據(jù)屬性 路徑 展示排序過濾商品融合1、商品: Len

11、ovo拯救者Y7000.2、商品: Lenovo330C.3、商品: Lenovo K5.21金融風(fēng)控ID1ID3ID2ID唯一性檢查循環(huán)擔(dān)保檢測二度關(guān)系觸黑黑用戶風(fēng)險評分?jǐn)U散1. 每個頂點一個組2. 將頂點加入鄰居 標(biāo)簽數(shù)目最多的組3. 組信息不再變化LPA 算法Louvain 算法屬性特征檢測關(guān)系特征檢測團伙檢測22HDFSHivePaloHDFSHiveKafkaMySQLColumn PartitionIT資產(chǎn)IDC百度云HadoopAFS業(yè)務(wù)組織移動安全AI安全安全SDK手機衛(wèi)士人員張三KafkaHiveHDFSETLEdge Vertex 大數(shù)據(jù)安全治理數(shù)據(jù)資產(chǎn)資產(chǎn) 視 圖數(shù) 據(jù)

12、血 緣數(shù)據(jù)安全隱私保護資產(chǎn)管理數(shù)據(jù)開發(fā)HugeGraph李四23王二HugeGraph存儲原理存儲結(jié)構(gòu)：邊集數(shù)組、鄰接矩陣、鄰接表、十字鏈表HugeGraph存儲結(jié)構(gòu)：鄰接表（順序表）24圖存儲結(jié)構(gòu)CSVHDFSJDBCGraph25Node邊集數(shù)組26邊ID源頂點目標(biāo)頂點邊屬性e1v1v2weighte2v1v3weighte3v2v6scoreae4v2v7scoree5v1v4weighte6v1v5weighte7v2v3weighte8v5v4weight172345e16 e2e5e6e3e4e7e8鄰接矩陣27v1v2v3v4v5v6v7v10123400v20070056v30

13、000000v40000900v50008000v60000000v7000000017234516 23456789鄰接表2817234516234567891110v1v2v3v5v2v6v7v3v3v4v4v5v5v4v6v7v6v1頂點下條邊十字鏈表2917234516 2345678912v113142627v2v3v4v5v615v72354 45鄰接表（順序表）30v3v4v5v6v7v7v2v1v2v5v5v4v1v417234516234567891110v6v3v1v2v4v1v5v2v4v3v2v6v1v3 v7出邊入邊如何參與開源貢獻Issue 問題與需求反饋Pull Request 貢獻代碼與評審31問題與需求反饋hps:/hugegraph/hugegraph/issues32貢獻代碼與評審1 fork2 git clone6 git push7 create prReview9 approveCode8 reviewrequest changes3 git checkout -b4 x c