EMC Isilon橫向擴(kuò)展存儲(chǔ)解決方案

云計(jì)算

大數(shù)據(jù)EMCIsilon橫向擴(kuò)展存儲(chǔ)解決方案技術(shù)顧問經(jīng)理：楊峰Yang.feng有計(jì)算環(huán)境的挑戰(zhàn)計(jì)算系統(tǒng)以超級(jí)計(jì)算機(jī)、集群或者網(wǎng)格為主多核計(jì)算迅速發(fā)展，物理機(jī)能力越來越強(qiáng)，原有的資源管理與調(diào)度器基于物理機(jī)，對(duì)于調(diào)度準(zhǔn)確度要求更高靜態(tài)調(diào)度：基于系統(tǒng)負(fù)載狀態(tài)和作業(yè)信息將調(diào)度作業(yè)到物理機(jī)器上難以滿足突發(fā)、批處理、CPU密集型、數(shù)據(jù)密集型等不同類型的作業(yè)對(duì)于不同的物理資源的需求如果作業(yè)類型與機(jī)器資源不匹配，會(huì)導(dǎo)致作業(yè)運(yùn)行效率低。如果采用多隊(duì)列調(diào)度，又會(huì)導(dǎo)致隊(duì)列之間的共享問題計(jì)算應(yīng)用越來越多樣化，但是傳統(tǒng)調(diào)度系統(tǒng)的接口單一、不易擴(kuò)展，嚴(yán)重限制了應(yīng)用推廣基于集群或者網(wǎng)格構(gòu)建分布式“私有云”，以按需定制的模式提供服務(wù)，可滿足應(yīng)用日益增長(zhǎng)的多樣性和廣泛性的需求遺留程序越來越多，老操作系統(tǒng)和軟件與新硬件無法配合，全部移植幾乎不可能如不能兼容，原有寶貴的數(shù)據(jù)無法讀取和分析引入虛擬化技術(shù)現(xiàn)有存儲(chǔ)環(huán)境的挑戰(zhàn)存儲(chǔ)以高性能并行文件系統(tǒng)為主容量管理PB成為基本配置，10PB-20PB已經(jīng)非常常見文件管理文件數(shù)量越來越多，數(shù)千萬至10億級(jí)別設(shè)備老化硬盤、電源等部件頻繁出錯(cuò)，導(dǎo)致服務(wù)中斷元數(shù)據(jù)問題可靠性性能瓶頸擴(kuò)容問題大規(guī)模數(shù)據(jù)遷移引入云存儲(chǔ)技術(shù)！EMCIsilon對(duì)云存儲(chǔ)的需求海量數(shù)據(jù)存儲(chǔ)，輕松管理PB級(jí)乃至數(shù)十PB的存儲(chǔ)空間全局命名空間，所有應(yīng)用看到統(tǒng)一的文件系統(tǒng)視圖支持標(biāo)準(zhǔn)接口，應(yīng)用無需修改直接運(yùn)行讀寫性能優(yōu)異，聚合帶寬高達(dá)數(shù)GB乃至數(shù)十GB易于管理維護(hù)，無需中斷業(yè)務(wù)即可輕松實(shí)現(xiàn)動(dòng)態(tài)擴(kuò)展基于開放架構(gòu)，可以運(yùn)行于任何開放架構(gòu)的硬件之上多級(jí)數(shù)據(jù)冗余，支持硬件與軟件冗余保護(hù)，數(shù)據(jù)高可靠多級(jí)存儲(chǔ)備份，靈活支持SSD、SAS、SATA、磁帶庫統(tǒng)一管理提綱EMCIsilon產(chǎn)品介紹EMCIsilon存儲(chǔ)解決方案總結(jié)內(nèi)容概括您將從中了解：EMCIsilon產(chǎn)品技術(shù)特性如果您的業(yè)務(wù)系統(tǒng)需要使用橫向擴(kuò)展架構(gòu)，Isilon哪個(gè)型號(hào)/功能是您所需要的大數(shù)據(jù)時(shí)代已經(jīng)到來地球物理

勘探醫(yī)學(xué)成像視頻

監(jiān)視移動(dòng)傳感器視頻渲染基因測(cè)序智能電網(wǎng)社交網(wǎng)絡(luò)電子支付大數(shù)據(jù)時(shí)代的橫向擴(kuò)展架構(gòu)VS縱向擴(kuò)展架構(gòu)縱向擴(kuò)展架構(gòu)橫向擴(kuò)展架構(gòu)Faster…………….Slower…………….TRAINSTOPS!!!!!!!!!!!!TRAINKEEPSGOING!!OneFS操作系統(tǒng)最佳的企業(yè)級(jí)大數(shù)據(jù)存儲(chǔ)——OneFS7.02012/11/26GA超大容量（20PB+）安全性高卓越性能數(shù)據(jù)保護(hù)可預(yù)期的性能互操作性操作簡(jiǎn)便應(yīng)對(duì)快速增長(zhǎng)EnterpriseSCALE-OUTCoreInnovation…

IsilonOneFSScale-Out系統(tǒng)單一文件系統(tǒng)，單一磁盤卷，最大支持20PB+大于80%的存儲(chǔ)利用率高性能，全對(duì)稱的集群系統(tǒng)易管理，易擴(kuò)展Multi-tier單一文件系統(tǒng)/單一集群

Scale-OutNASPerformanceOneFS7.0性能新特性106GB/s并發(fā)吞吐降低50%延遲724MB/s單流EnterpriseIT性能考量隨機(jī)/IO密集型BigData

性能考量吞吐1.6MIO/sSPECsfs2008

benchmark線性擴(kuò)展性

>20PB單一文件系統(tǒng)

(144nodes，4TBSATA盤)最大并發(fā)吞吐性能>=106GB/secIsilon典型架構(gòu)NFS,CIFS,FTP,HTTP客戶端/應(yīng)用端層標(biāo)準(zhǔn)的網(wǎng)絡(luò)層千兆/萬兆Isilon集群存儲(chǔ)層集群存儲(chǔ)內(nèi)部通訊InfiniBand層WindowsUNIX/LINUXMAC(optional2ndswitchforadditionalsubnet)(optional2ndswitchforhighavailability)Isilon產(chǎn)品家族Scale-outNASproductfamily性能和容量同時(shí)線性擴(kuò)展適用于IOPS密集型應(yīng)用的高性能解決方案S200高并發(fā)高吞吐的解決方案X200X400成本優(yōu)化的大容量擴(kuò)展解決方案NL400S-SeriesX-SeriesNL-SeriesIsilonS200配置可選項(xiàng)2RU2.5”10KRPMSASHDDDualIBPorts2x10GECard2x1GEportsVGA(HD15)IPMIServicePortConsolePort(DB9)USB(x2)PowerReceptaclesIsilonS200–前后視圖IsilonX200可選項(xiàng)配置:2RU3.5”7.2KRPMSATAHDDDualIBPorts2x10GECard2x1GEportsVGA(HD15)PowerReceptacles(Nema5-15R)IPMIServicePortConsolePort(DB9)USB(x2)IsilonX200–前后視圖IsilonX400可選項(xiàng)配置:IsilonX400–后視圖1243567812435678PowersuppliesBatteriesPowerswitchUSBportsSerialportGigabitEthernetexternalnetworkportsOptionalexternalnetworkportsInfinibandinternalnetworkportsIsilonNL400可選項(xiàng)配置:IsilonPlatform擴(kuò)展節(jié)點(diǎn)

性能加速器2顆4核CPU8/32GB內(nèi)存無硬盤2個(gè)萬兆高速端口或2個(gè)千兆端口加入集群時(shí)間低于15秒用于增加集群對(duì)外服務(wù)的吞吐/IOPSIsilonPlatform擴(kuò)展節(jié)點(diǎn)

備份加速器2顆4核CPU8GB內(nèi)存無硬盤4個(gè)4GbFC端口支持NDMPV3/V4提供可高達(dá)1.7TB/小時(shí)的備份速度IsilonOneFS產(chǎn)品架構(gòu)OneFS為EMCIsilon專利的操作系統(tǒng)，負(fù)責(zé)IO調(diào)度，集群管理Smartconnect?提供應(yīng)用訪問負(fù)載均衡和容錯(cuò)SmartPools?提供自動(dòng)分層功能SnapshotIQ?提供本地?cái)?shù)據(jù)保護(hù)/恢復(fù)SyncIQ?提供存儲(chǔ)間的數(shù)據(jù)復(fù)制（本地/異地），保證業(yè)務(wù)連續(xù)性SmartQuotas?報(bào)告/管理存儲(chǔ)資源的使用，同時(shí)擁有精簡(jiǎn)供應(yīng)功能SmartLock?提供WORM技術(shù)InsightIQ?提供存儲(chǔ)性能報(bào)表和使用趨勢(shì)分析HDFS特性支持Hadoop大數(shù)據(jù)應(yīng)用AsperaforIsilon?適用于遠(yuǎn)程數(shù)據(jù)復(fù)制，提高鏈路帶寬利用率強(qiáng)大但簡(jiǎn)易——Scale-out60秒完成擴(kuò)展無停機(jī)時(shí)間Isilon–線性,可預(yù)期的可擴(kuò)展性根據(jù)需求增長(zhǎng)容量和性能無需超量購買存儲(chǔ)增加無需增加管理的時(shí)間和精力減少管理時(shí)間和成本高效的業(yè)務(wù)的成果

無需數(shù)據(jù)移植單一,可擴(kuò)展的卷可預(yù)期的SLATheAnatomyofaWriteServers(optional2ndswitch)NFS,SMB,HDFSFTP,HTTPServersServersRAM(Cache)WriteCoalescerL2CacheNVRAMInfinibandHardDriveWritesarecacheduntilwritecoalescerisfull,timelimitreached,orprotocolrequestsconfirmationofdeliveryAtthesametime,datablocksassignedtoothernodesgothroughIBintotheirL2cacheOnceallnodeshaveallthedataandparityblocksinNVRAM,confirmationfromallnodesissenttothenodebeingwrittento,andacommitisreturnedtotheclientDataisthenwrittenontothespindlesWriteplanandparitycalculatedDatablock(s)assignedtothisnodestaycachedinL2forfuturereadsofthatfileDataisthencopiedintobatterysuppliedNVRAMfromL2CacheTheAnatomyofaReadServers(optional2ndswitch)NFS,SMB,HDFSFTP,HTTPServersServersRAM(Cache)L1CacheL2CacheNVRAMInfinibandHardDriveIfdataisn’talreadyinL2cache,datablockiscopiedfromtheHDDintoL2cache,thenIntoL1cacheDataisthenreconstructedInL1cacheRAM(Cache)L1CacheL2CacheNVRAMInfinibandHardDriveInthisexamplethough,sincewerecentlywrotethedata,itwasalreadyinL2cache.Wedon’tneedtoloaditfromtheHDD.ItjustloadsinfromtheL2cacheintotheL1cacheNode1Node2Node3DataLayoutDetailExample:3Node,N+1,1024KBfile,Concurrent1024KBFile128KB128KB128KB128KB128KB128KB128KB128KB128KB128KB128KB2+1FECProtectionGroups(Stripes)128KB128kDataStripeUnit128kProtectionStripeUnitNode1Node2Node3DataLayoutDetailExample:3Node,N+1,1024KBfile,Streaming1024KBFile128KB128KB128KB128KB128KB128KB128KB128KB128KB128KB128KB2+1FECProtectionGroups(Stripes)128KB128kDataStripeUnit128kProtectionStripeUnitNode1Node2Node3DataLayoutDetailExample:3Node,N+2:1,1024KBfile,Concurrent1024KBFile128KB128KB128KB128KB128KB128KB128KB128KB128KB128KB128KB2+1FECProtectionGroups(Stripes)128KBStripedepthisdoubledBlockswithinthesameStripenowarewrittentoseparatedrivesoneachnode128kDataStripeUnit128kProtectionStripeUnitOneFSFlexProtect–磁盤重構(gòu)數(shù)據(jù)ECC校驗(yàn)碼分布在各個(gè)節(jié)點(diǎn)的各個(gè)磁盤中相對(duì)傳統(tǒng)存儲(chǔ)，橫向擴(kuò)展架構(gòu)有更多的CPU和磁盤參與數(shù)據(jù)重構(gòu)Isilon傳統(tǒng)存儲(chǔ)16hourrebuild1hourrebuild最可靠的存儲(chǔ)系統(tǒng)——FlexProtect10GigEdedicatedAcceleratorXnodesNFSFailoverrequiredProcessingITFullaccess,maintenanceinterfaceCorporateDNS,noSCStatic(well-known)IPsrequiredBizzDevRenamedsub-domainCIFSclients(staticIP)10.10EngSharedsubnetSeparatesub-domainNFSFailoverVLAN(confidentialtraffic,isolated)SamephysicalLAN10.30FinanceStoragenodesNFSclients,nofailoverEditors10.20ServiceIPDNSDelegationRecordIsilonNodeIPBalancePolicySmartConnectzonenameDynamicIPs252627282930313226X1GigE10GigE應(yīng)用負(fù)載均衡——SmartConnect數(shù)據(jù)分層存放FAST_POOLOLD_DATASSD/SASDiskpoolSATADiskpool數(shù)據(jù)分層存放新文件寫入存儲(chǔ)30天之后…文件分布到FAST_POOL數(shù)據(jù)塊自動(dòng)遷移到OLD_POOL37配額管理及精簡(jiǎn)供應(yīng)——SmartQuotas基于目錄的Quota基于用戶的Quota基于組的QuotaThinProvisionSnapshotIQ

快照/恢復(fù)快速有效簡(jiǎn)單業(yè)務(wù)連續(xù)性解決方案——SnapshotIQ病毒意外刪除(optional2ndswitch)ServersNFS,CIFS,FTP,HTTP(optional2ndswitch)ServersServers備份服務(wù)器SAN交換機(jī)磁帶機(jī)FCAccelerator備份服務(wù)器上可運(yùn)行的備份軟件業(yè)務(wù)連續(xù)性解決方案——NDMP備份業(yè)務(wù)連續(xù)性解決方案——SyncIQ北京上海本地/遠(yuǎn)程數(shù)據(jù)復(fù)制基于IP鏈路改善數(shù)據(jù)鏈路復(fù)制效率一對(duì)多/多對(duì)一方式增強(qiáng)RPO/RTO異步復(fù)制方式正向/反向復(fù)制一鍵式恢復(fù)Failover/FailbackSIMPLE多認(rèn)證域的支持例如：整合法律部和財(cái)務(wù)部;財(cái)務(wù)部擁護(hù)登錄到LDAP1；法律部擁護(hù)登錄到LDAP3；存儲(chǔ)端數(shù)據(jù)各自獨(dú)立，互相無法訪問；SmartConnectZone1SystemZoneSmartConnectZone2LDAP1ActiveDirectory2LDAP3NISNISLocalUserGroupDatabaseAuthZone1AuthZone2FinanceLegalLDAP2ActiveDirectory1GroupDatabase對(duì)虛擬化的支持CIFSNFSFTPHTTPHDFS

(Hadoop)VAAIVASAOneFSAPIVAAI–VstorageAPIsForArrayIntegrationVASA–VsphereAPIsForStorageAwareness可寫虛機(jī)克隆網(wǎng)絡(luò)延遲大大減小與Vmware完美整合

簡(jiǎn)單PlatformOneFSRESTAPI管理和整合更加容易統(tǒng)一化管理和整合第三方應(yīng)用靈活HadoopVS.Google技術(shù)架構(gòu)的比較數(shù)據(jù)結(jié)構(gòu)化管理組件：Hbase→BigTable并行計(jì)算模型：MapReduce→MapReduce分布式文件系統(tǒng)：HDFS→GFSHadoop缺少分布式鎖服務(wù)ChubbyZookeeper是Chubby的克隆，為Hadoop高可用性服務(wù)提供保障.HBaseMapReduceHDFSBigTableMapReduceGFSHadoop云計(jì)算應(yīng)用ChubbyGoogle云計(jì)算應(yīng)用StorageMapReduceCompute業(yè)界唯一的內(nèi)嵌HDFS的橫向擴(kuò)展存儲(chǔ)解決方案內(nèi)嵌支持HDFS管理簡(jiǎn)單

HDFSTechnologyChallengesofHadoopHadoopDASEnvironment1DedicatedStorageInfrastructureOne-offforHadooponly2SinglePointofFailureNameNode3LackingEnterpriseDataProtectionNoSnapshots,replication,backup4PoorStorageEfficiency3Xmirroring5FixedScalabilityRigidcomputetostorageratio6ManualImport/ExportNoprotocolsupportNameNodeTechnologyChallengesofHadoopHadoopDASEnvironment1DedicatedStorageInfrastructureOne-offforHadooponly2SinglePointofFailureNameNode3LackingEnterpriseDataProtectionNoSnapshots,replication,backup4PoorStorageEfficiency3Xmirroring5FixedScalabilityRigidcomputetostorageratio6ManualImport/ExportNoprotocolsupport1x1x2x2x3x2x3x3x1xNameNode1Scale-OutStoragePlatformMultipleapplications&workflows2NoSinglePointofFailureDistributedNameNode3End-to-EndDataProtectionSnapshotIQ,SyncIQ,NDMPBackup4Industry-LeadingStorageEfficiency>80%StorageUtilization5IndependentScalabilityAddcompute&storageseparately6Multi-ProtocolIndustrystandardprotocolsNFS,CIFS,FTP,HTTP,HDFSHDFSEMCIsilonHadoop解決方案提綱EMCIsilon產(chǎn)品介紹EMCIsilon垂直行業(yè)解決方案介紹總結(jié)能源行業(yè)UpstreamProcessesandIndustryApplicationPartnersExplorationDevelopmentProductionEngineeringSeismicacquisitionSeismicprocessingSeismic&GeologicalinterpretationReservoirModelingDrilling&testingProductionDevelopmentandOptimizationReservoirSimulationFacilities&ReservoirEngineeringDrillingprogramProduction

LandmarkProMAXSeisSpaceLDIParadigmEchosSchlumbergerOmegaLandmarkR5000SeisWorksStratWorksGeoProbeDecisionSpaceParadigmSeisEarthStratEarthSchlumbergerPetrelGeoFrameSMTKingdom

LandmarkNexusPowerGridVIPParadigmSKUASchlumbergerEclipseIntersectPetrel

LandmarkCompassAssetPlannerWellborePlannerParadigmSysdrillSchlumbergerPetrelDrillingLandmarkOpenWellsSperryDrillingParadigmSysdrillSchlumbergerOspreyOperation

LandmarkDecisionSpaceforProductionTOWSchlumbergerAvocetAutomatedIntelligentDataTieringEMCExplorationSolutionsSeismicprocessingSeismicInterpretationGeologicalInterpretationReservoirModelingReservoirSimulationPetroVault:IntelligentScale-outTier2StorageforArchiveandStagingStorageSolutionsforSeismicProcessingIntelligentScale-outTier1StorageforInterpretationandModelingMakesnapshotseasy

UsetheFUSEIMplug-intocreatePetrelsnapshotsfromthesimple-to-useOceanuserinterfaceAccessaSnapshotLibrary

Browseyourlibraryofsnapshots:InsidePetrelorwithinaweb-browserByprojectchronologySelectasnapshottosee:DetailedprojectattributesStatisticsandhistoricalinformation

OnlyEMCoffersaSnapshotLibraryFeaturewiththePetrelSnapshotplug-in云存儲(chǔ)+

EMCISILONSCALE-OUTNASJeetuPatel,SyncplicityGM,VPJeffSchultz,Syncplicity,MarketingVinodMuralidhar,IsilonProductMgmt10December2012Streamingrecordinglink:/isilon/ldr.php?AT=pb&SP=MC&rID=117802012&rKey=b053de0490a4422dDownloadrecordinglink:/isilon/lsr.php?AT=dw&SP=MC&rID=117802012&rKey=e7a70b1cd947b62eOUTSIDEFIREWALLIPHONEANDROIDIPADIPADMACBOOKAIRIPHONEACROSSALLDEVICESSYNCFILESITSecurityandControlSyncAnyFolderSYNCPLICITYDIFFERENCEDISTRIBUTIONGROUPFILEITSecurityandControlSyncandNotificationsReadOnlyPermissionsSYNCPLICITYDIFFERENCEISILONTOMORROW:

CloudService/OnPremStorageFILESAUTHENTICATIONSyncplicity”O(jiān)rchestration”AuthenticationMetaDataEncryptionKeysSyncplicity”StorageandCompute”FilesON-PREMISECLOUD媒體

TypicalArchitecture–BroadcastMedia&Entertainment

TypicalArchitecture–OriginStorageforContentDeliveryMedia&Entertainment–Streaming

TypicalArchitecture–Film-scanforArchivalandBroadcastMedia&Entertainment