大數(shù)據(jù)技術(shù)交流課件

大數(shù)據(jù)平臺(tái)技術(shù)交流吳敏達(dá)–資深技術(shù)顧問(wèn)大數(shù)據(jù)平臺(tái)技術(shù)交流吳敏達(dá)–資深技術(shù)顧問(wèn)2從各種各樣類(lèi)型的巨大數(shù)據(jù)中，快速獲得有價(jià)值信息的能力，就是大數(shù)據(jù)技術(shù)什么是大數(shù)據(jù)Variety:

管理復(fù)雜的多角度關(guān)系和非關(guān)系類(lèi)型的數(shù)據(jù)（你是否忽略利用的非結(jié)構(gòu)化數(shù)據(jù)進(jìn)行決策嗎）Velocity:流數(shù)據(jù)或者大量數(shù)據(jù)的移動(dòng)（你是否希望通過(guò)實(shí)時(shí)操作提供更好的結(jié)果）Volume:數(shù)據(jù)量從TB級(jí)到ZB級(jí)（你是否收集了所有數(shù)據(jù)，并在使用它嗎）Veracity：1/3的領(lǐng)導(dǎo)在做業(yè)務(wù)決策時(shí)候不相信獲得的信息2從各種各樣類(lèi)型的巨大數(shù)據(jù)中，快速獲得有價(jià)值信息的能力，就是2大數(shù)據(jù)參考架構(gòu)超越傳統(tǒng)的數(shù)據(jù)倉(cāng)庫(kù)概念流計(jì)算Internet級(jí)別傳統(tǒng)數(shù)據(jù)倉(cāng)庫(kù)In-MotionAnalyticsDataAnalytics,DataOperations&ModelBuildingResultsInternetScaleDatabase&WarehouseAt-RestDataAnalyticsResultsUltraLowLatencyResultsInfoSphereBigInsights傳統(tǒng)/關(guān)系型數(shù)據(jù)源非傳統(tǒng)/非關(guān)系型數(shù)據(jù)源傳統(tǒng)/關(guān)系型數(shù)據(jù)源非傳統(tǒng)/非關(guān)系型數(shù)據(jù)源大數(shù)據(jù)參考架構(gòu)超越傳統(tǒng)的數(shù)據(jù)倉(cāng)庫(kù)概念流計(jì)算Internet傳3Cloud|Mobile|Security

IBM大數(shù)據(jù)平臺(tái)和應(yīng)用框架通過(guò)可視化的方法采集、抽取、以及探查數(shù)據(jù)應(yīng)用加速器，加速應(yīng)用開(kāi)發(fā)，快速實(shí)現(xiàn)分析價(jià)值BI/Reporting BI/ReportingExploration/VisualizationFunctional

AppIndustry

AppPredictiveAnalyticsContentAnalyticsAnalyticApplications（分析應(yīng)用）IBMBigDataPlatform（大數(shù)據(jù)平臺(tái)）SystemsManagementApplications&DevelopmentVisualization

&Discovery分析流數(shù)據(jù)，以及在大數(shù)據(jù)的是誰(shuí)數(shù)據(jù)洞察數(shù)據(jù)管控（數(shù)據(jù)質(zhì)量、生命周期、……）低成本地分析PB級(jí)結(jié)構(gòu)化和非結(jié)構(gòu)化數(shù)據(jù)操作型數(shù)據(jù)或者歷史數(shù)據(jù)的，基于數(shù)據(jù)倉(cāng)庫(kù)內(nèi)嵌分析Accelerators（加速器）InformationIntegration&Governance信息整合和管控Hadoop

SystemStreamComputingDataWarehouseContextualDiscovery索引和聯(lián)邦的上下文相關(guān)分析Cloud|Mobile|SecurityIBM大4議程IBMhadoop平臺(tái)－BigInsightsIBM流計(jì)算－StreamsIBM數(shù)據(jù)倉(cāng)庫(kù)平臺(tái)－pureData基于大數(shù)據(jù)平臺(tái)的數(shù)據(jù)分析-DataExplorerIBM大數(shù)據(jù)優(yōu)勢(shì)匯總議程56ForresterWave關(guān)于大數(shù)據(jù)的報(bào)告6ForresterWave關(guān)于大數(shù)據(jù)的報(bào)告BigInsights企業(yè)版連接和集成StreamsNetezzaTextprocessingengineandlibraryJDBCFlume基礎(chǔ)架構(gòu)JaqlHivePigHBaseMapReduceHDFSZooKeeperIndexingLuceneAdaptiveMapReduceOozieTextcompressionEnhancedsecurityFlexiblescheduler可選

IBM產(chǎn)品分析和探查應(yīng)用

DB2BigSheetsWebCrawlerDistribfilecopyDBexportBoardreaderDBimportAdhocqueryMachinelearningDataprocessing...管理和開(kāi)發(fā)工具

管理控制臺(tái)Monitorclusterhealth,jobs,etc.Add/removenodesStart/stopservicesInspectjobstatusInspectworkflowstatusDeployapplicationsLaunchapps/jobsWorkwithdistribfilesystemWorkwithspreadsheetinterfaceSupportREST-basedAPI...REclipse開(kāi)發(fā)工具TextanalyticsMapReduceprogrammingJaql,Hive,PigdevelopmentBigSheetsplug-indevelopmentOozieworkflowgenerationIntegratedinstallerOpenSourceIBMIBMCognosBIBigSQLAcceleratorformachinedataanalysisAcceleratorforsocialdataanalysisGuardiumDataStageDataExplorerSqoopHCatalogGPFS–FPOBigInsights企業(yè)版連接和集成StreamsNet7BigInsights優(yōu)勢(shì)列表HighPerformance&AvailabilityGPFS-FPOAtleast2XfasterthanopensourceHadoop17xthroughputspeedupfordocumentindexlookupsFaultresistanceforRealTimeDataPOSIXAdaptiveMapReduceSQLInterface(BigSQL)IntegratedInstall&MgtConsolesSecurityLDAP+HighspeedLZOCompressionDevelopmentToolingenvironment,testing,andoptimizationWarehouseRDBMS&StreamsIntegrationSystemT–TextAnalyticsBlazingFast,UsesUnstructureddata–doesnotrequirestructuring,(MapReduce)CustomizedAnnotatorsBigSheetsInsightEngineforanalyticsonMassiveamountsofdatainBigInsights.PowerofMap/ReducewithinreachoftheBusinessprofessionalwithafamiliarSpreadsheet-likeenvironment.BuiltinVisualizationsSystemML–MachineLeaning(Watson)DirectlyimplementedMLalgorithmsonMapReduceDeepStatistical/MiningembeddedintoBigInsightsPlatformBigIndexDistributedindexingandsearchParallelindexingandsearch企業(yè)級(jí)別基礎(chǔ)設(shè)施企業(yè)級(jí)別分析能力BigInsights優(yōu)勢(shì)列表HighPerforman8GPFS-FPO與HDFS各項(xiàng)指標(biāo)對(duì)比BigInsightsGPFS-FPO開(kāi)源HDFS或其他方案健壯性無(wú)單點(diǎn)故障

99.99%NameNode存在單點(diǎn)故障數(shù)據(jù)一致性高數(shù)據(jù)可能會(huì)丟失可擴(kuò)展性數(shù)千節(jié)點(diǎn)，實(shí)測(cè)4000+數(shù)千節(jié)點(diǎn)POSIX兼容完全兼容有限數(shù)據(jù)管理能力安全、備份、快照、緩存、復(fù)制有限傳統(tǒng)應(yīng)用性能好,兼顧讀寫(xiě)性能隨機(jī)讀寫(xiě)性能差安全性支持ACL,

容量限制，安全認(rèn)證不支持GPFS-FPO與HDFS各項(xiàng)指標(biāo)對(duì)比BigInsights9IBMAdaptiveMapReduce提供強(qiáng)大的企業(yè)級(jí)管理，用于在可擴(kuò)展的共享網(wǎng)格上運(yùn)行分布式應(yīng)用程序和大數(shù)據(jù)分析。它可加速數(shù)十個(gè)并行應(yīng)用程序，以加快實(shí)現(xiàn)成果并更好地利用所有可用資源。TeraSortThroughputSWIM10timesfewerCPUcores6timesfaster60timesfasterBerkleySWIMisaworkloadbenchmarkdevelopedatUniversityofCaliforniaatBerkley.MeasurecoreschedulingefficiencyofMapReduceworkloadsatHadoopWorld2011Multi-tenantresourcemanagement10xLesshardwareforthefastestTeraSortscore.IBMAdaptiveMapReduce提供強(qiáng)大的企業(yè)10BigSQL:讓Hadoop原生支持SQL原生SQL支持BigInsightsANSISQL92+Standardsyntaxsupport(joins,datatypes,…)

真正的JDBC/ODBCPreparedstatementsCancelsupportDatabasemetadataAPIsupportSecuresocketconnections(SSL)優(yōu)化LeveragingMapReduceparallelism

or…Directaccessforlow-latencyqueries多種數(shù)據(jù)源HBase(includingsecondaryindexes)CSV,Delimitedfiles,SequencefilesJSONHivetablesBigSQLEngineBigInsightsDataSourcesSQLHiveTablesHBasetablesCSVFilesApplicationJDBC/ODBCServerJDBC/ODBCDriverBigSQL:讓Hadoop原生支持SQL原生11使用報(bào)表工具CognosBIserver可以下推計(jì)算到BigInsights更快響應(yīng)時(shí)間沒(méi)有Hive的限制Application

(Map-Reduce)Storage

(HBase,HDFS)InfoSphereBigInsightsCognosBIServerExplore&AnalyzeReport&ActSQLInterface

viaJDBC使用報(bào)表工具CognosBIserver可以下推計(jì)算到12可以使用已有的工具:SQuirreLSQLUsingexistingSQLtoolingagainstBigDataSupportfor“standard”authentication!!

(notsupportedforHive,butsupportedbyBigSQL!)13可以使用已有的工具:SQuirreLSQLUsinge13可以使用已有的工具:EclipseUsingexistingSQLtoolingagainstBigDataSamesetupasforexistingSQLsources!!Supportfor“standard”authentication!!14可以使用已有的工具:EclipseUsingexisti14集成的基于Web的安裝無(wú)縫的單節(jié)點(diǎn)或者集群模式安裝開(kāi)源組件和IBM組件的安裝驗(yàn)證檢查，確保系統(tǒng)正常運(yùn)行集成的基于Web的安裝15基于Web的管理控制平臺(tái)任務(wù)和工作流管理系統(tǒng)健康監(jiān)控集群以及文件系統(tǒng)管理基于Web的管理控制平臺(tái)任務(wù)和工作流管理16基于表單的分析基于表單的分析17儀表盤(pán)儀表盤(pán)18議程IBMhadoop平臺(tái)－BigInsightsIBM流計(jì)算－StreamsIBM數(shù)據(jù)倉(cāng)庫(kù)平臺(tái)－pureData基于大數(shù)據(jù)平臺(tái)的數(shù)據(jù)分析-DataExplorerIBM大數(shù)據(jù)優(yōu)勢(shì)匯總議程19計(jì)算模式的變革動(dòng)態(tài)數(shù)據(jù)的實(shí)時(shí)分析

流數(shù)據(jù)結(jié)構(gòu)或非結(jié)構(gòu)化動(dòng)態(tài)數(shù)據(jù)流流計(jì)算實(shí)時(shí)對(duì)流數(shù)據(jù)進(jìn)行分析計(jì)算靜態(tài)數(shù)據(jù)的歷史分析

批處理模式查詢(xún)驅(qū)動(dòng)：靜態(tài)數(shù)據(jù)提交查詢(xún)依靠數(shù)據(jù)庫(kù)，數(shù)據(jù)倉(cāng)庫(kù)傳統(tǒng)計(jì)算模式流計(jì)算模式QueriesMemoryDiskUpdatesMemoryDiskEventDataQueriesAlertsActions20計(jì)算模式的變革動(dòng)態(tài)數(shù)據(jù)的實(shí)時(shí)分析靜態(tài)數(shù)據(jù)的歷史分析

傳統(tǒng)計(jì)算IBM

InfoSphereStreamsAplatformforreal-timeanalyticsonBIGdata一個(gè)處理流數(shù)據(jù)的低延遲平臺(tái)毫秒級(jí)，甚至微妙級(jí)端到端的延遲一個(gè)可高度擴(kuò)展的，用于實(shí)時(shí)分析的高性能平臺(tái)通過(guò)橫向增加硬件獲得近線(xiàn)性的處理能力擴(kuò)展高達(dá)125個(gè)節(jié)點(diǎn)擴(kuò)展一個(gè)靈活的、動(dòng)態(tài)的平臺(tái)Streams應(yīng)用靈活部署支持動(dòng)態(tài)部署新的分析應(yīng)用

MillionsofeventspersecondMicrosecondLatencyTraditional/Non-traditionaldatasourcesRealtimedecisionsPowerfulAnalyticsAlgoTradingTelcochurnpredictSmartGridCyberSecurityGovernment/LawenforcementICUMonitoringEnvironmentMonitoring21IBMInfoSphereStreamsAplatfo連續(xù)注入連續(xù)分析實(shí)現(xiàn)可擴(kuò)展：將應(yīng)用分布到多個(gè)計(jì)算節(jié)點(diǎn)

在流連接的硬件節(jié)點(diǎn)之間分發(fā)Streams流計(jì)算平臺(tái)基礎(chǔ)設(shè)施提供服務(wù)：

在跨硬件/軟件節(jié)點(diǎn)中調(diào)度分析

建立流媒體連接

…變換過(guò)濾/采樣分類(lèi)關(guān)聯(lián)注釋在適當(dāng)?shù)牡胤剑?處理單元可以是“融合”在一起 從而消除通信的延遲連續(xù)注入連續(xù)分析實(shí)現(xiàn)可擴(kuò)展：Streams流計(jì)算平臺(tái)基StreamsToolkit(常用)JoinFunctorAggregatePunctorSortFilterDirectory-ScanFileSourceFileSinkUDPSourceUDPSinkTCPSourceTCPSinkExportImportODBCSourceODBCEnrichsolidDBEnrichInetSourceODBCAppendStreamsToolkit(常用)JoinFuncto高度可并行擴(kuò)展能力同時(shí)利用幾十臺(tái)、上百臺(tái)進(jìn)行實(shí)時(shí)數(shù)據(jù)流處理高度可并行擴(kuò)展能力同時(shí)利用幾十臺(tái)、上百臺(tái)進(jìn)行實(shí)時(shí)數(shù)據(jù)流處理24X86BoxX86BladeCellBladeBlueGeneFPGABladeX86BladeX86BladeX86BladeX86BladeX86BladeTransportStreamsDataFabric高可用、集群部署能力ProcessingElementContainerProcessingElementContainerProcessingElementContainerProcessingElementContainerProcessingElementContainer優(yōu)化調(diào)度器把操作符分配到不同的節(jié)點(diǎn)運(yùn)行，并持續(xù)監(jiān)控資源的使用情況自適應(yīng)資源、工作負(fù)載、數(shù)據(jù)速率的變化在低成本硬件上運(yùn)行－從單節(jié)點(diǎn)PC到刀片服務(wù)器到多陣列集群X86X86BladeCellBlueGeneFPGA2526參考性能數(shù)據(jù)

－吞吐性能和延遲1,975streams2,133streams163streams24channels163DecisionEngines356BlueGeneNodes356ProcessingElements4,274streamsDataFeed每秒500萬(wàn)條記錄平均延遲150微秒最小延遲50微秒65K中49條延遲>2毫秒26參考性能數(shù)據(jù)－吞吐性能和延遲1,975stream26大數(shù)據(jù)的實(shí)時(shí)分析平臺(tái)

分析不同的數(shù)據(jù)源，每秒分析上百萬(wàn)個(gè)事件．易用性能圖形化應(yīng)用開(kāi)發(fā)模式，容易管理和監(jiān)控．集成能力集成XML,MQ,DataStage,HDFS等．高級(jí)工具包和加速器事件序列和地理信息數(shù)據(jù)庫(kù)工具箱和CEP處理

可定制化的電信和媒體分析加速器幫助快速部署應(yīng)用.InfoSphereStreamsInfoSphereStreams

大數(shù)據(jù)的實(shí)時(shí)分析平臺(tái)InfoSphereStreamsIn議程IBMhadoop平臺(tái)－BigInsightsIBM流計(jì)算－StreamsIBM數(shù)據(jù)倉(cāng)庫(kù)平臺(tái)－pureData基于大數(shù)據(jù)平臺(tái)的數(shù)據(jù)分析-DataExplorerIBM大數(shù)據(jù)優(yōu)勢(shì)匯總議程28

－專(zhuān)家集成數(shù)據(jù)系統(tǒng)forTransactions為滿(mǎn)足交易型數(shù)據(jù)服務(wù)而優(yōu)化的系統(tǒng)forOperationalAnalytics為滿(mǎn)足操作型分析數(shù)據(jù)服務(wù)而優(yōu)化的系統(tǒng)forAnalytics為滿(mǎn)足分析型數(shù)據(jù)服務(wù)而優(yōu)化的系統(tǒng)DataPlatformDeliveringDataServices－專(zhuān)家集成數(shù)據(jù)系統(tǒng)f29議程IBMhadoop平臺(tái)－BigInsightsIBM流計(jì)算－StreamsIBM數(shù)據(jù)倉(cāng)庫(kù)平臺(tái)－pureData基于大數(shù)據(jù)平臺(tái)的數(shù)據(jù)分析-DataExplorerIBM大數(shù)據(jù)優(yōu)勢(shì)匯總議程30NamechangeandnewreleaseIBMInfoSphereDataExplorerNamechangeandnewreleaseIBM31Application/Users結(jié)合企業(yè)各類(lèi)數(shù)據(jù)，包括大數(shù)據(jù)的分析結(jié)果FileSystemsRelationalDataContentManagementEmailCRMSupplyChainERPRSSFeedsExternalSourcesCloudCustomSourcesVelocityPlatformIBMBigDataPlatformSystemsManagementApplicationDevelopmentVisualization

&DiscoveryAcceleratorsInformationIntegration&GovernanceHadoop

SystemStreamComputingDataWarehouseCommentingRatingSharedFoldersTaggingApplication/結(jié)合企業(yè)各類(lèi)數(shù)據(jù)，包括大數(shù)據(jù)的分析結(jié)IDE給最終用戶(hù)提供整合的企業(yè)級(jí)的信息探查ApplicationFrameworkRealTimeAnalyticsInternetScaleAnalyticsIn-DatabaseAnalyticsFederatedDiscoveryNavigationandVisualization33EnterpriseDataConnectorsIDE給最終用戶(hù)提供整合的企業(yè)級(jí)的信息探查Applicati33IDE

－InfoSphereDataExplorer平臺(tái)架構(gòu)ConnectorFrameworkCM,RM,DMRDBMSFeedsWeb2.0EmailWebCRM,ERPFileSystemsApplicationFrameworkAuthentication/AuthorizationQueryTransformationPersonalizationDisplayUserProfilesFederatedSourcesSubscriptionsFeedsWebResultsOtherAppsApplicationApplicationApplicationThesaurusClusteringOntologySupportSemanticProcessingEntityExtractionRelevancyTextAnalyticsFacetingTaggingTaxonomyCollaborationMetaDataConvertingSearchEngineCrawlingIndexingIDE－InfoSphereDataExplorer平議程IBMhadoop平臺(tái)－BigInsightsIBM流計(jì)算－StreamsIBM數(shù)據(jù)倉(cāng)庫(kù)平臺(tái)－pureData基于大數(shù)據(jù)平臺(tái)的數(shù)據(jù)分析-DataExplorerIBM大數(shù)據(jù)優(yōu)勢(shì)匯總議程35IBMBigData方案的價(jià)值提供最為廣泛的平臺(tái)支持-解決了海量數(shù)據(jù)處理的多種需求IBM研究和研發(fā)實(shí)驗(yàn)室的先進(jìn)數(shù)據(jù)處理技術(shù)融入到產(chǎn)品當(dāng)中接受，拓展，延伸并反饋到開(kāi)源社區(qū)產(chǎn)品支持和保持領(lǐng)先的專(zhuān)業(yè)知識(shí)與能力，以確?？蛻?hù)的成功處理海量數(shù)據(jù)的完整技術(shù)支持其他的友商只能提供部分的支持，或通過(guò)其他廠商的方案進(jìn)行融合專(zhuān)注于企業(yè)品質(zhì)和訴求經(jīng)過(guò)實(shí)踐的性能管理與開(kāi)發(fā)工具支持IBM內(nèi)部和IBM外部的信息管理技術(shù)的深度整合安全性和數(shù)據(jù)治理高可用與備份恢復(fù)IBM的服務(wù)和咨詢(xún)IBM的先進(jìn)技術(shù)和長(zhǎng)久積累下的經(jīng)驗(yàn)沉淀.IBMBigData方案的價(jià)值提供最為廣泛的平臺(tái)支持-解36謝謝謝謝37大數(shù)據(jù)平臺(tái)技術(shù)交流吳敏達(dá)–資深技術(shù)顧問(wèn)大數(shù)據(jù)平臺(tái)技術(shù)交流吳敏達(dá)–資深技術(shù)顧問(wèn)39從各種各樣類(lèi)型的巨大數(shù)據(jù)中，快速獲得有價(jià)值信息的能力，就是大數(shù)據(jù)技術(shù)什么是大數(shù)據(jù)Variety:

管理復(fù)雜的多角度關(guān)系和非關(guān)系類(lèi)型的數(shù)據(jù)（你是否忽略利用的非結(jié)構(gòu)化數(shù)據(jù)進(jìn)行決策嗎）Velocity:流數(shù)據(jù)或者大量數(shù)據(jù)的移動(dòng)（你是否希望通過(guò)實(shí)時(shí)操作提供更好的結(jié)果）Volume:數(shù)據(jù)量從TB級(jí)到ZB級(jí)（你是否收集了所有數(shù)據(jù)，并在使用它嗎）Veracity：1/3的領(lǐng)導(dǎo)在做業(yè)務(wù)決策時(shí)候不相信獲得的信息2從各種各樣類(lèi)型的巨大數(shù)據(jù)中，快速獲得有價(jià)值信息的能力，就是39大數(shù)據(jù)參考架構(gòu)超越傳統(tǒng)的數(shù)據(jù)倉(cāng)庫(kù)概念流計(jì)算Internet級(jí)別傳統(tǒng)數(shù)據(jù)倉(cāng)庫(kù)In-MotionAnalyticsDataAnalytics,DataOperations&ModelBuildingResultsInternetScaleDatabase&WarehouseAt-RestDataAnalyticsResultsUltraLowLatencyResultsInfoSphereBigInsights傳統(tǒng)/關(guān)系型數(shù)據(jù)源非傳統(tǒng)/非關(guān)系型數(shù)據(jù)源傳統(tǒng)/關(guān)系型數(shù)據(jù)源非傳統(tǒng)/非關(guān)系型數(shù)據(jù)源大數(shù)據(jù)參考架構(gòu)超越傳統(tǒng)的數(shù)據(jù)倉(cāng)庫(kù)概念流計(jì)算Internet傳40Cloud|Mobile|Security

IBM大數(shù)據(jù)平臺(tái)和應(yīng)用框架通過(guò)可視化的方法采集、抽取、以及探查數(shù)據(jù)應(yīng)用加速器，加速應(yīng)用開(kāi)發(fā)，快速實(shí)現(xiàn)分析價(jià)值BI/Reporting BI/ReportingExploration/VisualizationFunctional

AppIndustry

AppPredictiveAnalyticsContentAnalyticsAnalyticApplications（分析應(yīng)用）IBMBigDataPlatform（大數(shù)據(jù)平臺(tái)）SystemsManagementApplications&DevelopmentVisualization

&Discovery分析流數(shù)據(jù)，以及在大數(shù)據(jù)的是誰(shuí)數(shù)據(jù)洞察數(shù)據(jù)管控（數(shù)據(jù)質(zhì)量、生命周期、……）低成本地分析PB級(jí)結(jié)構(gòu)化和非結(jié)構(gòu)化數(shù)據(jù)操作型數(shù)據(jù)或者歷史數(shù)據(jù)的，基于數(shù)據(jù)倉(cāng)庫(kù)內(nèi)嵌分析Accelerators（加速器）InformationIntegration&Governance信息整合和管控Hadoop

SystemStreamComputingDataWarehouseContextualDiscovery索引和聯(lián)邦的上下文相關(guān)分析Cloud|Mobile|SecurityIBM大41議程IBMhadoop平臺(tái)－BigInsightsIBM流計(jì)算－StreamsIBM數(shù)據(jù)倉(cāng)庫(kù)平臺(tái)－pureData基于大數(shù)據(jù)平臺(tái)的數(shù)據(jù)分析-DataExplorerIBM大數(shù)據(jù)優(yōu)勢(shì)匯總議程4243ForresterWave關(guān)于大數(shù)據(jù)的報(bào)告6ForresterWave關(guān)于大數(shù)據(jù)的報(bào)告BigInsights企業(yè)版連接和集成StreamsNetezzaTextprocessingengineandlibraryJDBCFlume基礎(chǔ)架構(gòu)JaqlHivePigHBaseMapReduceHDFSZooKeeperIndexingLuceneAdaptiveMapReduceOozieTextcompressionEnhancedsecurityFlexiblescheduler可選

IBM產(chǎn)品分析和探查應(yīng)用

DB2BigSheetsWebCrawlerDistribfilecopyDBexportBoardreaderDBimportAdhocqueryMachinelearningDataprocessing...管理和開(kāi)發(fā)工具

管理控制臺(tái)Monitorclusterhealth,jobs,etc.Add/removenodesStart/stopservicesInspectjobstatusInspectworkflowstatusDeployapplicationsLaunchapps/jobsWorkwithdistribfilesystemWorkwithspreadsheetinterfaceSupportREST-basedAPI...REclipse開(kāi)發(fā)工具TextanalyticsMapReduceprogrammingJaql,Hive,PigdevelopmentBigSheetsplug-indevelopmentOozieworkflowgenerationIntegratedinstallerOpenSourceIBMIBMCognosBIBigSQLAcceleratorformachinedataanalysisAcceleratorforsocialdataanalysisGuardiumDataStageDataExplorerSqoopHCatalogGPFS–FPOBigInsights企業(yè)版連接和集成StreamsNet44BigInsights優(yōu)勢(shì)列表HighPerformance&AvailabilityGPFS-FPOAtleast2XfasterthanopensourceHadoop17xthroughputspeedupfordocumentindexlookupsFaultresistanceforRealTimeDataPOSIXAdaptiveMapReduceSQLInterface(BigSQL)IntegratedInstall&MgtConsolesSecurityLDAP+HighspeedLZOCompressionDevelopmentToolingenvironment,testing,andoptimizationWarehouseRDBMS&StreamsIntegrationSystemT–TextAnalyticsBlazingFast,UsesUnstructureddata–doesnotrequirestructuring,(MapReduce)CustomizedAnnotatorsBigSheetsInsightEngineforanalyticsonMassiveamountsofdatainBigInsights.PowerofMap/ReducewithinreachoftheBusinessprofessionalwithafamiliarSpreadsheet-likeenvironment.BuiltinVisualizationsSystemML–MachineLeaning(Watson)DirectlyimplementedMLalgorithmsonMapReduceDeepStatistical/MiningembeddedintoBigInsightsPlatformBigIndexDistributedindexingandsearchParallelindexingandsearch企業(yè)級(jí)別基礎(chǔ)設(shè)施企業(yè)級(jí)別分析能力BigInsights優(yōu)勢(shì)列表HighPerforman45GPFS-FPO與HDFS各項(xiàng)指標(biāo)對(duì)比BigInsightsGPFS-FPO開(kāi)源HDFS或其他方案健壯性無(wú)單點(diǎn)故障

99.99%NameNode存在單點(diǎn)故障數(shù)據(jù)一致性高數(shù)據(jù)可能會(huì)丟失可擴(kuò)展性數(shù)千節(jié)點(diǎn)，實(shí)測(cè)4000+數(shù)千節(jié)點(diǎn)POSIX兼容完全兼容有限數(shù)據(jù)管理能力安全、備份、快照、緩存、復(fù)制有限傳統(tǒng)應(yīng)用性能好,兼顧讀寫(xiě)性能隨機(jī)讀寫(xiě)性能差安全性支持ACL,

容量限制，安全認(rèn)證不支持GPFS-FPO與HDFS各項(xiàng)指標(biāo)對(duì)比BigInsights46IBMAdaptiveMapReduce提供強(qiáng)大的企業(yè)級(jí)管理，用于在可擴(kuò)展的共享網(wǎng)格上運(yùn)行分布式應(yīng)用程序和大數(shù)據(jù)分析。它可加速數(shù)十個(gè)并行應(yīng)用程序，以加快實(shí)現(xiàn)成果并更好地利用所有可用資源。TeraSortThroughputSWIM10timesfewerCPUcores6timesfaster60timesfasterBerkleySWIMisaworkloadbenchmarkdevelopedatUniversityofCaliforniaatBerkley.MeasurecoreschedulingefficiencyofMapReduceworkloadsatHadoopWorld2011Multi-tenantresourcemanagement10xLesshardwareforthefastestTeraSortscore.IBMAdaptiveMapReduce提供強(qiáng)大的企業(yè)47BigSQL:讓Hadoop原生支持SQL原生SQL支持BigInsightsANSISQL92+Standardsyntaxsupport(joins,datatypes,…)

真正的JDBC/ODBCPreparedstatementsCancelsupportDatabasemetadataAPIsupportSecuresocketconnections(SSL)優(yōu)化LeveragingMapReduceparallelism

or…Directaccessforlow-latencyqueries多種數(shù)據(jù)源HBase(includingsecondaryindexes)CSV,Delimitedfiles,SequencefilesJSONHivetablesBigSQLEngineBigInsightsDataSourcesSQLHiveTablesHBasetablesCSVFilesApplicationJDBC/ODBCServerJDBC/ODBCDriverBigSQL:讓Hadoop原生支持SQL原生48使用報(bào)表工具CognosBIserver可以下推計(jì)算到BigInsights更快響應(yīng)時(shí)間沒(méi)有Hive的限制Application

(Map-Reduce)Storage

(HBase,HDFS)InfoSphereBigInsightsCognosBIServerExplore&AnalyzeReport&ActSQLInterface

viaJDBC使用報(bào)表工具CognosBIserver可以下推計(jì)算到49可以使用已有的工具:SQuirreLSQLUsingexistingSQLtoolingagainstBigDataSupportfor“standard”authentication!!

(notsupportedforHive,butsupportedbyBigSQL!)50可以使用已有的工具:SQuirreLSQLUsinge50可以使用已有的工具:EclipseUsingexistingSQLtoolingagainstBigDataSamesetupasforexistingSQLsources!!Supportfor“standard”authentication!!51可以使用已有的工具:EclipseUsingexisti51集成的基于Web的安裝無(wú)縫的單節(jié)點(diǎn)或者集群模式安裝開(kāi)源組件和IBM組件的安裝驗(yàn)證檢查，確保系統(tǒng)正常運(yùn)行集成的基于Web的安裝52基于Web的管理控制平臺(tái)任務(wù)和工作流管理系統(tǒng)健康監(jiān)控集群以及文件系統(tǒng)管理基于Web的管理控制平臺(tái)任務(wù)和工作流管理53基于表單的分析基于表單的分析54儀表盤(pán)儀表盤(pán)55議程IBMhadoop平臺(tái)－BigInsightsIBM流計(jì)算－StreamsIBM數(shù)據(jù)倉(cāng)庫(kù)平臺(tái)－pureData基于大數(shù)據(jù)平臺(tái)的數(shù)據(jù)分析-DataExplorerIBM大數(shù)據(jù)優(yōu)勢(shì)匯總議程56計(jì)算模式的變革動(dòng)態(tài)數(shù)據(jù)的實(shí)時(shí)分析

流數(shù)據(jù)結(jié)構(gòu)或非結(jié)構(gòu)化動(dòng)態(tài)數(shù)據(jù)流流計(jì)算實(shí)時(shí)對(duì)流數(shù)據(jù)進(jìn)行分析計(jì)算靜態(tài)數(shù)據(jù)的歷史分析

批處理模式查詢(xún)驅(qū)動(dòng)：靜態(tài)數(shù)據(jù)提交查詢(xún)依靠數(shù)據(jù)庫(kù)，數(shù)據(jù)倉(cāng)庫(kù)傳統(tǒng)計(jì)算模式流計(jì)算模式QueriesMemoryDiskUpdatesMemoryDiskEventDataQueriesAlertsActions57計(jì)算模式的變革動(dòng)態(tài)數(shù)據(jù)的實(shí)時(shí)分析靜態(tài)數(shù)據(jù)的歷史分析

傳統(tǒng)計(jì)算IBM

InfoSphereStreamsAplatformforreal-timeanalyticsonBIGdata一個(gè)處理流數(shù)據(jù)的低延遲平臺(tái)毫秒級(jí)，甚至微妙級(jí)端到端的延遲一個(gè)可高度擴(kuò)展的，用于實(shí)時(shí)分析的高性能平臺(tái)通過(guò)橫向增加硬件獲得近線(xiàn)性的處理能力擴(kuò)展高達(dá)125個(gè)節(jié)點(diǎn)擴(kuò)展一個(gè)靈活的、動(dòng)態(tài)的平臺(tái)Streams應(yīng)用靈活部署支持動(dòng)態(tài)部署新的分析應(yīng)用

MillionsofeventspersecondMicrosecondLatencyTraditional/Non-traditionaldatasourcesRealtimedecisionsPowerfulAnalyticsAlgoTradingTelcochurnpredictSmartGridCyberSecurityGovernment/LawenforcementICUMonitoringEnvironmentMonitoring58IBMInfoSphereStreamsAplatfo連續(xù)注入連續(xù)分析實(shí)現(xiàn)可擴(kuò)展：將應(yīng)用分布到多個(gè)計(jì)算節(jié)點(diǎn)

在流連接的硬件節(jié)點(diǎn)之間分發(fā)Streams流計(jì)算平臺(tái)基礎(chǔ)設(shè)施提供服務(wù)：

在跨硬件/軟件節(jié)點(diǎn)中調(diào)度分析

建立流媒體連接

…變換過(guò)濾/采樣分類(lèi)關(guān)聯(lián)注釋在適當(dāng)?shù)牡胤剑?處理單元可以是“融合”在一起 從而消除通信的延遲連續(xù)注入連續(xù)分析實(shí)現(xiàn)可擴(kuò)展：Streams流計(jì)算平臺(tái)基StreamsToolkit(常用)JoinFunctorAggregatePunctorSortFilterDirectory-ScanFileSourceFileSinkUDPSourceUDPSinkTCPSourceTCPSinkExportImportODBCSourceODBCEnrichsolidDBEnrichInetSourceODBCAppendStreamsToolkit(常用)JoinFuncto高度可并行擴(kuò)展能力同時(shí)利用幾十臺(tái)、上百臺(tái)進(jìn)行實(shí)時(shí)數(shù)據(jù)流處理高度可并行擴(kuò)展能力同時(shí)利用幾十臺(tái)、上百臺(tái)進(jìn)行實(shí)時(shí)數(shù)據(jù)流處理61X86BoxX86BladeCellBladeBlueGeneFPGABladeX86BladeX86BladeX86BladeX86BladeX86BladeTransportStreamsDataFabric高可用、集群部署能力ProcessingElementContainerProcessingElementContainerProcessingElementContainerProcessingElementContainerProcessingElementContainer優(yōu)化調(diào)度器把操作符分配到不同的節(jié)點(diǎn)運(yùn)行，并持續(xù)監(jiān)控資源的使用情況自適應(yīng)資源、工作負(fù)載、數(shù)據(jù)速率的變化在低成本硬件上運(yùn)行－從單節(jié)點(diǎn)PC到刀片服務(wù)器到多陣列集群X86X86BladeCellBlueGeneFPGA6263參考性能數(shù)據(jù)

－吞吐性能和延遲1,975streams2,133streams163streams24channels163DecisionEngines356BlueGeneNodes356ProcessingElements4,274streamsDataFeed每秒500萬(wàn)條記錄平均延遲150微秒最小延遲50微秒65K中49條延遲>2毫秒26參考性能數(shù)據(jù)－吞吐性能和延遲1,975stream63大數(shù)據(jù)的實(shí)時(shí)分析平臺(tái)

分析不同的數(shù)據(jù)源，每秒分析上百萬(wàn)個(gè)事件．易用性能圖形化應(yīng)用開(kāi)發(fā)模式，容易管理和監(jiān)控．集成能力集成XML,MQ,DataStage,HDFS等．高級(jí)工具包和加速器事件序列和地理信息數(shù)據(jù)庫(kù)工具箱和CEP處理

可定制化的電信和媒體分析加速器幫助快速部署應(yīng)用.InfoSphereStreamsInfoSphereStreams

大數(shù)據(jù)的實(shí)時(shí)分析平臺(tái)InfoSphereStreamsIn議程IBMhadoop平臺(tái)－BigInsightsIBM流計(jì)算－StreamsIBM數(shù)據(jù)倉(cāng)庫(kù)平臺(tái)－pureData基于大數(shù)據(jù)平臺(tái)的數(shù)據(jù)分析-DataExplorerIBM大數(shù)據(jù)優(yōu)勢(shì)匯總議程65

－專(zhuān)家集成數(shù)據(jù)系統(tǒng)forTransactions為滿(mǎn)足交易型數(shù)據(jù)服務(wù)而優(yōu)化的系