大數(shù)據(jù)分析存儲(chǔ)解決方案

從企業(yè)數(shù)據(jù)向大數(shù)據(jù)的擴(kuò)展TraditionalApproachStructured,analytical,logicalSystemsofRecordNewApproach

Creative,holisticthought,intuitionSystemsOfEngagementMultimediaSystemsofInsightEnterpriseIntegration

andContextAccumulationStructured

Repeatable

LinearUnstructured

Exploratory

DynamicDataWarehouseWebLogsSocialDataTextData:

emailsSensordata:

imagesRFIDInternalAppDataTransactionDataMainframeDataOLTPSystemDataHadoopand

StreamsTraditionalSourcesNewSourcesERP

data具備洞悉才干的系統(tǒng)SystemsofInsight對(duì)新式根底架構(gòu)的需求在可靠和平安的環(huán)境中處置關(guān)鍵業(yè)務(wù)運(yùn)用存取和處置海量數(shù)據(jù)——包括構(gòu)造化和非構(gòu)造化數(shù)據(jù)速度及時(shí)呼應(yīng)隨時(shí)能夠出現(xiàn)的商業(yè)時(shí)機(jī)，這就需求靈敏、實(shí)時(shí)性的根底架構(gòu)ThedynamicsofSoRandSoE:經(jīng)過(guò)負(fù)載及資源部署的優(yōu)化，來(lái)加強(qiáng)靈敏性和效益經(jīng)過(guò)采用包括基于開(kāi)放規(guī)范的技術(shù)等新技術(shù)來(lái)改善ITeconomicsSystemofRecord〔SoR〕SystemsofEngagement〔SoE〕對(duì)的決策對(duì)的地方對(duì)的時(shí)間點(diǎn)BigData&Analytics大數(shù)據(jù)分析的新型架構(gòu)處理方案IBMBigData&AnalyticsInfrastructureDataZoneApplicationZone4SmartMeteringGridOperations電網(wǎng)管理FieldService外勤現(xiàn)場(chǎng)效力ResourcePlanning資源規(guī)劃CustomerService/CustomerOperations實(shí)現(xiàn)真正的有效的法規(guī)服從及時(shí)發(fā)現(xiàn)能源損耗問(wèn)題、以及偷電和欺詐行為提高客戶稱心度電量運(yùn)用預(yù)測(cè)更為準(zhǔn)確電網(wǎng)運(yùn)維優(yōu)化減少停電次數(shù)和時(shí)間案例:SmartMetering智慧電力計(jì)費(fèi)大數(shù)據(jù)分析運(yùn)用可以帶來(lái)真正的業(yè)務(wù)價(jià)值法規(guī)服從案例:用大數(shù)據(jù)分析來(lái)加強(qiáng)SmartMetering數(shù)據(jù)分析的高可用性，以確保隨時(shí)了解用戶喜好跨運(yùn)用的TB級(jí)的數(shù)據(jù)需求–通用虛擬化存儲(chǔ)平臺(tái)實(shí)時(shí)搜集、存儲(chǔ)并分析數(shù)據(jù)，最快可達(dá)50,000datapoints/sec歷史用電形狀數(shù)據(jù)的復(fù)雜查詢處置數(shù)據(jù)在加載到數(shù)據(jù)倉(cāng)庫(kù)前的清洗、驗(yàn)證，這些數(shù)據(jù)能夠來(lái)自很多的用戶、收費(fèi)系統(tǒng)或斷電維護(hù)系統(tǒng)關(guān)系掌控

構(gòu)建和維護(hù)電網(wǎng)的獨(dú)一試圖對(duì)整個(gè)企業(yè)的構(gòu)造化和非構(gòu)造化數(shù)據(jù)t做全局導(dǎo)覽Navigation，從中發(fā)現(xiàn)Discover價(jià)值分析用戶用電情況，偵測(cè)偷電、改表等行為預(yù)測(cè)哪些用戶適宜于哪些分時(shí)時(shí)段電價(jià)或需求/呼應(yīng)效力分時(shí)時(shí)段電價(jià)的實(shí)時(shí)定價(jià)或

提供及時(shí)的需求/呼應(yīng)效力IBMBigData&AnalyticsReferenceArchitectureBigDataPlatformCapabilitiesInformationIngestReal-timeAnalyticsWarehouse&DataMartsAnalyticAppliancesAllDataSourcesAdvancedAnalytics/

NewInsightsNew/

EnhancedApplicationsCognitive認(rèn)知LearnDynamically?Prescriptive規(guī)范BestOutcomes?Predictive預(yù)測(cè)WhatCouldHappen?Descriptive

描畫WhatHasHappened?ExplorationandDiscoveryWhatDoYouHave?StreamingDataTextDataApplicationsDataTimeSeriesGeoSpatialRelationalSocialNetworkVideo&ImageAutomatedProcessCaseManagementAnalyticApplicationsWatsonCloudServicesISVSolutionsAlertsNewInfrastructureLeveragesDataTypesDatain

MotionDataat

RestDatain

ManyFormsInformationIngestionandOperationalInformationDecision

ManagementBIandPredictiveAnalyticsNavigation

andDiscoveryIntelligence

AnalysisRawDataStructuredDataTextAnalyticsDataMiningEntityAnalyticsMachineLearningLandingArea,AnalyticsZoneandArchiveVideo/AudioNetwork/SensorEntityAnalyticsPredictiveReal-timeAnalyticsExploration,IntegratedWarehouse,andMartZonesDiscoveryDeepReflectionOperationalPredictiveStreamProcessingDataIntegrationMasterDataStreamsInformationGovernance,SecurityandBusinessContinuityBigInsightsStreamsWarehouseInfoSphereBigInsightsHadoop-based低延遲分析，針對(duì)多樣化的、海量靜態(tài)數(shù)據(jù)Data-At-RestNetezzaHighCapacityAppliance基于構(gòu)造化數(shù)據(jù)的可查詢歸檔Netezza1000基于構(gòu)造化數(shù)據(jù)的

BI+定制化分析DataSmartAnalyticsSystem基于構(gòu)造化數(shù)據(jù)的運(yùn)營(yíng)分析InformixTimeseriesTime-structuredanalyticsInfoSphereWarehouse基于構(gòu)造化數(shù)據(jù)的大容量數(shù)據(jù)分析InfoSphereStreams低延遲流數(shù)據(jù)分析Velocity,Variety&VolumeData-In-MotionMPPDataWarehouseStreamComputingInformationIntegrationHadoopInfoSphereInformationServer海量數(shù)據(jù)集成和轉(zhuǎn)化ApacheHadoop:跨效力器集群的大數(shù)據(jù)集分布式處置開(kāi)放系統(tǒng)框架，采用的是一種簡(jiǎn)單化編程模型IBMBigDataPlatform大數(shù)據(jù)平臺(tái)What:一種開(kāi)源軟件，將數(shù)據(jù)計(jì)算分布到整個(gè)集群的常見(jiàn)商用效力器和存儲(chǔ)上Why:傳統(tǒng)的計(jì)算架構(gòu)是一種沿縱向擴(kuò)展方式，經(jīng)過(guò)更快的SAN、大容量?jī)?nèi)存和多級(jí)緩存將數(shù)據(jù)加載到CPU上，本錢比較高。What:Hadoop把大數(shù)據(jù)集合拆分區(qū)劃為小數(shù)據(jù)集合，再把小數(shù)據(jù)集合分發(fā)到多臺(tái)普通效力器上，是一種橫向擴(kuò)展方式。Why:Scalable,Flexible,CostEffective,FaultTolerentComponents:MapReduce,HDFSWhatisHadoop?NameNode(Metadatastore)NodesHDFSClusterOperatingSystemNodesElasticStorage-SNCClusterKernelLevelIBMValueforHadoop!HDFS把數(shù)據(jù)分散存儲(chǔ)在多個(gè)存儲(chǔ)節(jié)點(diǎn)Node上HDFS設(shè)計(jì)時(shí)就假設(shè)存儲(chǔ)節(jié)點(diǎn)有失效的能夠，所以HDFS會(huì)把一份數(shù)據(jù)復(fù)制3份以上，分散存儲(chǔ)在多個(gè)節(jié)點(diǎn)上，從而實(shí)現(xiàn)系統(tǒng)整體上的可靠性HDFS文件系統(tǒng)是由效力器節(jié)點(diǎn)集群組成的，每臺(tái)效力器按照HDFS的特有block協(xié)議支持網(wǎng)絡(luò)化block數(shù)據(jù)HDFSNameNode有發(fā)生單點(diǎn)缺點(diǎn)的危險(xiǎn)IBM在改善文件系統(tǒng)的性能同時(shí)消除了單點(diǎn)缺點(diǎn)——ElasticStorage-SNC(availableasbetacode)Hadoop闡明,MapReduce,HDFSHadoopStackWhatdoesitlooklike?典型Hadoop存儲(chǔ)的PainPoints在選擇HDFS的組件〔如軟件、效力器、網(wǎng)絡(luò)和存儲(chǔ)等〕時(shí)很難選對(duì)在從測(cè)試環(huán)境遷移到消費(fèi)環(huán)境時(shí)，需求做的調(diào)優(yōu)和調(diào)整任務(wù)太繁復(fù)了長(zhǎng)期繼續(xù)不斷的運(yùn)維保證過(guò)于繁重，比如老要改換失效組件〔尤其是硬盤〕，這使得保證期望的SLA非常難CPU和存儲(chǔ)去耦本來(lái)用戶的CPU和內(nèi)存曾經(jīng)滿足計(jì)算需求，但為了存儲(chǔ)容量需求安裝更多的硬盤不得不買更多的、不用要的CPU和內(nèi)存Storageoptionsavailablehavecleargaps本地存儲(chǔ)的利用率低(~25%)，每次需求擴(kuò)容的時(shí)候就要添加更多的效力器，而一旦硬盤失效后需求重建，效力器越多，失效的幾率越高，性能也就越差I(lǐng)BMStorageforHadoop傳統(tǒng)的Hadoop集群運(yùn)用的是效力器內(nèi)置硬盤存儲(chǔ)。假設(shè)用作測(cè)試或科學(xué)研討還好，可作為業(yè)務(wù)運(yùn)轉(zhuǎn)的存儲(chǔ)就要采用企業(yè)存儲(chǔ)Hadoop集群要擔(dān)任數(shù)據(jù)維護(hù)和復(fù)制重建〔就是copy〕失效的數(shù)據(jù)集到不同節(jié)點(diǎn)上——嚴(yán)重影響CPU性能，無(wú)法實(shí)現(xiàn)企業(yè)級(jí)的RASReplicatedata–問(wèn)題同上擴(kuò)展的時(shí)候同時(shí)添加處置器/網(wǎng)絡(luò)/存儲(chǔ)，無(wú)法做到物盡其用〔nowaytoseparatethese3evenifexcesscapacityexistinginone(e.g.NeededmorestoragebuthadtoaddComputeandNetwork)〕運(yùn)用外部存儲(chǔ)可以將存儲(chǔ)負(fù)載和Hadoop計(jì)算節(jié)點(diǎn)分別，同時(shí)還獲得了企業(yè)存儲(chǔ)的益處。SellthevalueofXIV,V7000,SVC,etc.用戶普通會(huì)隨HadoopFileSystem部署；采用ElasticStorage可以有很多益處14數(shù)據(jù)加速ExperiencetheinstantresultsthatcomefromIBMFlashSystemDriveasmuchas45Xfasteranalyticsresultsoncertainworkloads數(shù)據(jù)負(fù)載的多樣性和靈敏性XIVdeliverspredictableperformancethatscaleslinearlywithouthotspotsdeliveringinsightsfromanalyticsfasterwithtuning-freedatadistributionScale-out,parallelprocessingofElasticStoragesoftwareandintegrationwithFlashSystemdramaticallyacceleratesperformanceofAnalyticsclustersVirtualStorageCenterwithSVCautomaticallyoptimizesdatawarehouseperformanceandcostacrossFlashandDiskMainframeDataEnvironmentsIntegrationwithDB2&specialtyanalytics“engines〞leveragingDS8870delivers4xreductioninbatchtimeswithnewHighPerformanceFlashEnclosuresHighspeedencryptiononeverydrivetypesecuresdata數(shù)據(jù)維護(hù)和保管LTFSEEw/tapeprovidesreducedTCObyupto90%overdiskforlongtermretentionofdataatrestwithalargeopenformattaperepositoryReducetheamountofdatatobestoredbyupto25timeswithProtecTIERde-duplication12x更快IBMFlashSystemincreasedSPLUNK&SASapplicationefficiencytoperformbusinessanalytics20x改善inactionablesupplychainanalytics,4xreductioninbatchtimes,virtualizationforplug&play6x時(shí)間節(jié)省“GPFSallowsustomovethemetadatafromthedisktotheFlashSystemonline.Oncewedidthat,thebackupswerereduceddowntoaboutanhour.〞2hrsbecomes2minutes失效切換時(shí)間大幅縮短MappingCharacteristicstoIBMStorageProductsStorageInfrastructure需求適用于一切的5種運(yùn)用場(chǎng)景OptimizedMulti-TemperatureWarehouse優(yōu)化的多級(jí)存儲(chǔ)庫(kù)AllFlashFlashSystemHybridDS8000EasyTierXIV+SSDCachingStorwizeEasyTierFlashSystemSolution(VSC+FlashSystem)PureSystemsPureFlex(XIVorStorwizew/EasyTier)PureDataforTransactions(Storwize)PureDataforAnalytics(Netezza)Midrange&EntryTier0AccelerationSmarterStorageIntegratedSystemsEnterpriseOfferingsXIVzEnterpriseSolutionsforAnalyticswithDS8000PureDataSystemforOperationalAnalyticswithStorwizePureFlexSystemwithStorwizeDS8000SmartAnalyticsSystemswithDS3xxxOpen&ExtensibleStorwizefamilyFlashSystemfamilyIBMSmarterStorage的設(shè)計(jì)就是支持大數(shù)據(jù)分析

高效和優(yōu)化數(shù)據(jù)根底架構(gòu)IBMFlashSystem：為大數(shù)據(jù)分析運(yùn)用設(shè)計(jì)的，讓運(yùn)用和數(shù)據(jù)實(shí)現(xiàn)極速IBMFlashSystem的極速性能讓實(shí)時(shí)業(yè)務(wù)決策成為能夠適宜于模塊化數(shù)據(jù)存儲(chǔ)構(gòu)造的Hadoop系統(tǒng)。某些或一切數(shù)據(jù)可以保管到Flash閃存上，其他可以保管到XIVIBMXIV:OptimizeddataworkloaddiversityforBigData&AnalyticsIBMXIV的高性能無(wú)須人工干涉配置，且適用于各種各樣的存儲(chǔ)負(fù)載IBMXIV的效率高的異乎尋常，而且簡(jiǎn)單性業(yè)內(nèi)最高，內(nèi)置友好界面IBMXIV的彈性是企業(yè)級(jí)的，完全保證了數(shù)據(jù)的可用性和業(yè)務(wù)延續(xù)性XIV:為Analytics而生無(wú)與倫比的性能可擴(kuò)展的網(wǎng)格存儲(chǔ)架構(gòu)恣意時(shí)間支持恣意讀寫負(fù)載板上的閃存Flash

無(wú)與倫比的可靠性精致的數(shù)據(jù)分布無(wú)雙的磁盤重建時(shí)間企業(yè)級(jí)的可用性

無(wú)與倫比的簡(jiǎn)易性簡(jiǎn)單的規(guī)劃、供應(yīng)和靈敏性上線后零維護(hù)零調(diào)優(yōu)“XIV最吸引我們的地方就是其超強(qiáng)的性能…we正是由于XIV為我們的精細(xì)復(fù)雜的分析運(yùn)用提供了一致的高性能，使得我們可以為我們的用戶帶來(lái)更多的價(jià)值。〞SAS和XIV網(wǎng)格架構(gòu)——完美的結(jié)合大規(guī)模并行計(jì)算堅(jiān)持繼續(xù)地最正確性能BalancedPerformance性能平衡年年零調(diào)整UnprecedentedScalability史無(wú)前例的擴(kuò)展性配合添加SAS節(jié)點(diǎn)和XIV模塊即可IBMSVC:OptimizeddataworkloadflexibilityforBigData&AnalyticsIBMSVC經(jīng)過(guò)如下功能在IBM大數(shù)據(jù)產(chǎn)品線上添加了靈敏性：完好和數(shù)據(jù)虛擬化和數(shù)據(jù)挪動(dòng)性高級(jí)集群和復(fù)制多路鏡像，readpreferredoptionRealTimeCompression實(shí)時(shí)緊縮EasyTierHotExtentcachingStorwizeV7000/UIBMSVC設(shè)計(jì)原那么Real-TimeCompression實(shí)時(shí)緊縮是設(shè)計(jì)來(lái)做：作用于ActivePrimaryData公用的緊縮平臺(tái)PlatformhandlesALLheavyliftingassociatedwithcompression不會(huì)影響性能Wemodifyacompressedfilein-placeefficiently不會(huì)改動(dòng)用戶運(yùn)用Usersnoradminsneedtochangeanything處置流程不變緊縮是在線完成，不是事后緊縮業(yè)界規(guī)范緊縮算法所采用的緊縮算法曾經(jīng)運(yùn)用了幾十年StorwizeV7000/UIBMSVC24流處置計(jì)算&IBMFlashSystemsData:是擁有還是保管?或是是分析和開(kāi)場(chǎng)行動(dòng)!DatainDataat25InfoSphereStreams:大數(shù)據(jù)流分析為分析動(dòng)態(tài)數(shù)據(jù)而建多并發(fā)輸入數(shù)據(jù)流大規(guī)?？蓴U(kuò)展Massivescalability分析和處置的數(shù)據(jù)多樣化Structured,unstructured,video,audioAdvancedanalyticoperators自順應(yīng)實(shí)時(shí)分析WithDataWarehousesWithHadoopSystemsCurrentfactfinding當(dāng)前數(shù)據(jù)查詢分許流動(dòng)中的數(shù)據(jù)——在數(shù)據(jù)落盤前低延遲方式,pushmodel數(shù)據(jù)驅(qū)動(dòng)——真正的數(shù)據(jù)分析Historicalfactfinding歷史數(shù)據(jù)查詢查找和分析存儲(chǔ)在磁盤上的數(shù)據(jù)信息批處置方式,pullmodel查詢驅(qū)動(dòng):submitsqueriestostaticdataTraditionalComputingStreamComputing流數(shù)據(jù)計(jì)算代表著計(jì)算方式的變化Real-timeAnalyticsRealTimeAnalytics實(shí)時(shí)分析

想象一下他如何用防火栓喝水來(lái)自多個(gè)多樣輸入源的大量數(shù)據(jù)直接處置和過(guò)濾數(shù)據(jù)，而不用存儲(chǔ)僅保管有價(jià)值的數(shù)據(jù)僅關(guān)聯(lián)對(duì)數(shù)據(jù)最感興趣的用戶隨著數(shù)據(jù)信息的產(chǎn)生采取行動(dòng)AdaptiveAnalytics自順應(yīng)分析

DatainMotionandDataatRest的集成1.DataIngest數(shù)據(jù)集成，數(shù)據(jù)發(fā)掘，機(jī)器學(xué)習(xí)，統(tǒng)計(jì)建模實(shí)時(shí)和歷史數(shù)據(jù)洞察力的可視化3.AdaptiveAnalyticsModel數(shù)據(jù)收取，

在線分析預(yù)備，方式校驗(yàn)Data2.Bootstrap/EnrichControlflowInfoSphereBigInsights,Database&WarehouseInfoSphereStreams

AdaptiveReal-TimeAnalytics自順應(yīng)實(shí)時(shí)分析來(lái)自多個(gè)多樣輸入源的大量數(shù)據(jù)過(guò)去、如今和未來(lái)全方位綜合性視圖實(shí)時(shí)分析，低延時(shí)結(jié)果Fullcontextfordeepanalysis深度分析的完好的上下文跨datainmotionanddataatrest的常用數(shù)據(jù)分析自順應(yīng)-隨機(jī)而變當(dāng)發(fā)現(xiàn)非預(yù)期行為時(shí)，自順應(yīng)當(dāng)識(shí)別出新數(shù)據(jù)意義時(shí)深度分析之開(kāi)場(chǎng)沒(méi)有認(rèn)識(shí)到的數(shù)據(jù)意義，隨后才能夠認(rèn)識(shí)到自順應(yīng)——在開(kāi)場(chǎng)沒(méi)有認(rèn)識(shí)到的，隨后可以找出數(shù)據(jù)方式StockmarketImpactofweatheronsecuritiespricesAnalyzemarketdataatultra-lowlatenciesMomentumCalculatorFraudpreventionDetectingmulti-partyfraudRealtimefraudpreventione-ScienceSpaceweatherpredictionDetectionoftransienteventsSynchrotronatomicresearchGenomicResearchTransportationIntelligenttrafficmanagementAutomotiveTelematicsEnergy&UtilitiesTransactivecontrolPhasorMonitoringUnitDownholesensormonitoringNaturalSystemsWildfiremanagementWatermanagementOtherManufacturingTextAnalysisERPforCommoditiesReal-timemultimodalsurveillanceSituationalawarenessCybersecuritydetectionLawEnforcement,

Defense&CyberSecurityHealth&LifeSciencesICUmonitoringEpidemicearlywarningsystemRemotehealthcaremonitoringTelephonyCDRprocessingSocialanalysisChurnpredictionGeomapping如何運(yùn)用InfoSphereStreams?加快數(shù)據(jù)流入分析系統(tǒng)的速度向買賣方向加速。。。一個(gè)高效和靈敏的根底架構(gòu)顯然可以加快流速，并平衡不同數(shù)據(jù)分析的需求CoresSCMStorageNetworkCoresSCMStorageNetworkCoresSCMStorageNetworkCoresSCMStorageNetwork++預(yù)測(cè)分析

數(shù)據(jù)倉(cāng)庫(kù)文本分析HadoopWorkloads優(yōu)化敏感性分析加快流速價(jià)值時(shí)間“觸發(fā)事件〞數(shù)據(jù)完備買賣Insight預(yù)見(jiàn)獲取數(shù)據(jù)時(shí)間分析數(shù)據(jù)時(shí)間行動(dòng)時(shí)間大數(shù)據(jù)分析的新式根底架構(gòu)處理方案IBMBigData&AnalyticsInfrastructureDataZoneApplicationZoneExperiencereal-timeanalyticalinsightswithupto50xbetterperformancethanenterprisedisksystemsusingIBMFlashCore?technologyPreserveandprotectinfrastructurecontinuitywhilescalingtoover2petabyteofeffectiveall-flashcapacityunderasingleintegrateinterfaceDeliveragilityanddataeconomicswith4xgreatercapacityinlessrackspacethancompetitiveall-flashproductsSynchronizedandComplimentarytoOverarchingStorageMessaging-Acceleratetimetoinsightsthrough"datawithoutborders."IBMinnovationfreesdatawithagileandsimpletousestoragesolutionsdeliveringsuperiordataeconomicsIBMFlashSystemCoreLaunchMessagingDriveacompleteparadigmshiftinEnterpriseStoragewiththeallnewIBMFlashSystemFamilyIBMFlashSystemFamily

2021ThemeTimetoinsight.Timetovalue.Timetomarket.IBMFlashSystem,it’sabouttime.FlashRealized!IBMFlashSystemV9000

FoundationalPillarsIBMFlashCore?TechnologyistheDNAoftheFlashSystemFamilyScalablePerformanceEnduringEconomicsAgileIntegrationIntroducingtheNewIBMFlashSystemFamilyOfferingsIBMFlashSystem900ExtremePerformance:Delivers100microsecondresponsetimesMacroEfficiency:Lowestlatencyofferingwith>40%greatercapacityatalowercostpercapacityEnterpriseReliability:IBMenhancedMicronMLCflashtechnologywithFlashWearGuaranteePoweredbyIBMFlashCore?TechnologyIBMFlashSystemV9000ScalablePerformance:Growcapacityandperformancewithupto2.2PBscalingcapabilityEnduringEconomics:NextgenerationflashmediawithlowercostpercapacityAgileIntegration:FullyintegratedsystemmanagementtosimplifymanagementandimproveworkforceproductivityunderasinglenamespaceFlashSystem900IntroducingIBMFlashSystem900,thenextgenerationinourlowestlatencyofferingIBMMicroLatency?withupto1.1millionIOPS40%greatercapacityata10%lowercostpercapacityIBMFlashCore?technology,oursecretsauceTechnicalcollaborationwith