邵賽賽周康：AGI時代統(tǒng)一數(shù)據(jù)目錄的設計與實踐

演講人：邵賽賽演講人：周康ApacheGravitino創(chuàng)始人 實驗的跟蹤和可視化打包、運行ML流程作業(yè)超參優(yōu)化打包、運行ML流程作業(yè)超參優(yōu)化評估GenAI應用訓練，調優(yōu)，及生產(chǎn)化監(jiān)控、調試LLM評估GenAI應用訓練，調優(yōu)，及生產(chǎn)化開發(fā)GenAI應用AutomationAutomation自動觸發(fā)流程發(fā)布和分享模型及數(shù)據(jù)集自動觸發(fā)流程ReportsArtifactsDataReportsQ不同組織之間分享模型、數(shù)據(jù)集數(shù)據(jù)集、模型生不同組織之間分享模型、數(shù)據(jù)集數(shù)據(jù)集、模型生命周期管理數(shù)據(jù)集、模型治理及權限控制https://www.llamaindex.ai/blog/introducing-llamacloud-and-llamaparse-af8cedf9006b下------------、karkakarka數(shù)據(jù)連接ConnecttotheDataThatMatters數(shù)據(jù)發(fā)現(xiàn)Automaticallyfind,classify,數(shù)據(jù)連接ConnecttotheDataThatMatters數(shù)據(jù)發(fā)現(xiàn)Automaticallyfind,classify,andmapallofyourdata-everywhere.數(shù)據(jù)主權Automateend-to-enddatarightsrequestsandreporting.3rdParty數(shù)據(jù)主權Automateend-to-enddatarightsrequestsandreporting.數(shù)據(jù)分類AutomaticallyAutomaticallyclassifymoretypesofdatainmoreplaces.元數(shù)據(jù)增強Enrichtechnicalmetadatawith元數(shù)據(jù)增強Enrichtechnicalmetadatawithbusinessandoperationalmetadataforfullvisibility.數(shù)據(jù)生命周期管理Simplifyandautomatedatalifecyclemanagementfromcollectiontodestruction.–HiveHiveMetastoreBuilt-inBuilt-inCatalogSchemaSchemaRegistryRegistryRegistry消息隊列數(shù)據(jù)倉庫數(shù)據(jù)湖機器學習消息隊列數(shù)據(jù)倉庫數(shù)據(jù)湖機器學習●數(shù)據(jù)統(tǒng)一視圖●從元數(shù)據(jù)層面上達到SSOT●統(tǒng)一訪問和治理Next-GenDataCatalogistheCoreinNewOpenUnifiedAccessingUnifiedManagingGravitinoUnifiedAccessingUnifiedManagingGravitinoWarehouseGravitinoGravitinoGravitinoGravitinoFunctionalitylayerInterfacelayerCorewithobjectmodelConnectionLayerMetadataMetadataStorageUnifiedUnifiedProcessingUnifiedGoverningUnifiedUnifiedRESTAPIsIcebergIcebergRESTAPIsCatalogCatalogCatalogSchemaSchemaSchemaTableModelFilesetMetalakeCatalogCatalogCatalogSchemaSchemaSchemaTableModelFilesetMetalakeCatalogCatalogSchemaSchemaTopicTopicAlterAlter…}….}ArrowFileSystemArrowFileSystemAlterADLSADLS…}storageLocation:str….}Spark/Spark/Flink/Presto對接統(tǒng)一元數(shù)據(jù)完善用戶審計轉換底層權限隱藏認證憑據(jù)在線數(shù)據(jù)庫數(shù)據(jù)湖表格OLAP系統(tǒng)對接機器學習框架傳統(tǒng)機器學習對接機器學習框架傳統(tǒng)機器學習大模型預訓練/微調流程打通、產(chǎn)品打通DataOps/MLOps/LLMOps非結構化數(shù)據(jù)資產(chǎn)地圖與資產(chǎn)治理存量數(shù)據(jù)納管支持設置TTL與TTV數(shù)據(jù)處理流程AI流程數(shù)據(jù)加工spark/Flink數(shù)據(jù)分析OLAP數(shù)據(jù)探查數(shù)據(jù)加工spark/Flink數(shù)據(jù)分析OLAP數(shù)據(jù)探查Notebook模型訓練LLM模型部署數(shù)據(jù)集成集成引擎根據(jù)血緣推薦TTL、TTV管理前降本40%管理后特征分析特征分析pytorch模型訓練pandaspytorch模型訓練TalosModel模型部署數(shù)據(jù)落倉提取特征模型部署數(shù)據(jù)落倉lcebergFileset模型校驗lcebergFilesetModelModel數(shù)據(jù)分析