ShigekiMisawaRHICComputingFacilityBrookhavenNational

1、Shigeki MisawaRHIC Computing FacilityBrookhaven National LaboratoryFacility EvolutionChange Drivers at the FacilityUsersComponent CostsTechnologySecurityOperational ExperienceUser DesiresUnlimited on line storage spaceUniform, global Infinite bandwidthUnlimited I/O transaction ratesInfinite processi

2、ng powerNo learning curveHigh availability$0 costCurrent ConfigurationCurrent Configuration (contd)HPSS managed tape archive10 IBM Server9840/9940 tape drivesNFS Servers14 E4505 V480100TB RAID5MTI and Zyzzx(CMD)Brocade SANGigE BackboneAlcatel (PacketEngine)CiscoAlteon (phased out)SysKonnect NIC1000

3、Dual CPU Linux nodes w/local scratch2 AFS CellsIBM AFS 3.6.xCurrent Configuration (contd)Secure GatewaysSsh/bbftpFacility FirewallSoftware for optimized from HPSSCustom and LFS batch controlLimited management of NFS/local scratch spaceOther experiment specific middle-wareMatching User DesiresHPSS pr

4、ovides near infinite storage space, although not on lineNFS provides large amounts of on line storage, uniform, global namespaceLinux Farm provides significant amount of processing power.Relatively low costIssuesHigh bandwidth ?Transaction rate ?High availability ?Learning curve ?User Processing Mod

5、elsReconstructionProcessing models among users are similarProcessing is well definedAnalysisWide range of processing styles in useTransitioning to finite set of processing models difficult to institute.Requires high level experiment acceptanceRequires adoption of processing models by individual user

6、sSecurity PerspectiveFacility FirewallOn and Off site considered hostileGateways (ssh, bbftp) bypass FW, provide access with no clear text passwordsExposure is getting smallerManagement and monitoring getting betterPrimary vulnerablitiesUser lifecycle managementReusable passwordsDistributed Changes

7、affecting securityRigorous DOE mandated user life cycle managementKerberos 5 and LDAP authorization/authenticationTighter network security (inbound/outbound access)Grid integrationWeb servicesGrid daemonsDeployment vs maintenanceHPSSProvides virtually unlimited storage abilityP inconvenient and can

8、be problematicGeneral user access to HSI, a good thing ?Work aroundsLimit direct accessIdentify and eliminate problematic accessUtilize optimization toolsPrognosis:Good enough, too much invested to switchGigabit/100Mb EthernetGigE not a performance issueDriving it is another questionGigE port/switch

9、 costs an issueVendor shake out somewhat of an issueMix of copper/optical technology a nuisance100Mb compute node connectivity sufficient for the forseeable future Linux FarmAdequately provides needed processing powerSplit between pure batch and batch/interactive nodesNodes originally disk light, no

10、w disk heavyCheaper ($/MB) than RAID 5 NFS diskBetter performanceRobust with respect to NFS/HPSS hiccupsChanging processing modelLinux Farm LimitationIndividual nodes are now mission critical (since they are stateful)Maintenance an issue (HD must be easily s)Nodes no longer identicalNode lifecycle p

11、roblematicLocal disk space management issuesNon global namespace and access to dataEOL of CPU vs EOL of DiskNFS ServersProviding relatively large amounts of storage (100TB)Availability and reliability getting betterServers not a problemProblems with all disk components, RAID controllers, hubs, cable

12、s, disk drives, GBICs, configuration tools, monitoring tools, switch, .Overloaded servers are now the primary problemNFS Servers4x450MHz E-450, 2x900MHz V480 (coming on line soon)SysKonnect GigE NIC (Jumbo/non-jumbo)MTI and Zyzzx RAID 5 storageBrocade FabricVeritas VxFS/VxVMPerformance of NFS Server

13、sMaximum observed BW 55 MB/sec (non jumbo)NFS Logging recently enabled (and then disabled)Variable access patterns1.4TB/day max observed BW out of a serverMax MB transferred usually, not the most highly accessedData files accessed, but shared libraries and log files are also accessed.Limited statist

14、ics makes further conclusions difficult to makeNFS Servers and disks are poorly utilizedNFS Logging (Solaris)Potentially a useful toolWebalizer used to analyze resulting log filesDaily stats on hits and KB transferredTime, client, and Poor implementationGeneration of binary/text log files problemati

15、cBusy FS - observed 10MB/minute, 1-2 GB/hour log write rateUnder high load, nfslogd cannot keep up with binary nfslogd unable to analyze binary log files 1.5 GBnfslogd cannot be run offlineNFS OptimizationWithout usage statistics cannot tune statisticsSpace usageI/O transactions/FS BW / FSLoosely or

16、ganized user community makes tracking and controlling of user behavior difficultFuture Directions for NFS ServersContinued expansion of current architectureCurrent planReplace with fully distributed disksNeeds middleware (from grid?) to manage. Can users be taught to use the system ? Fewer but more

17、capable servers (i.e., bigger SMP servers)Not likely ($cost)Will performance increase ?Future Directions for NFS ServersMore, cheaper NFS appliancesCurrently not likely (administrative issues)IDE RAID systems ?Technological maturity/administrative issuesSpecialized CIF/NFS servers ?Cost/technologica

18、l maturity/administrative issuesOther technologies ?Cost/technological maturity/administrative issueAFS ServersCurrent State2 AFS CellsIBM AFS 3.6.x12 ServersSystem working adequatelyFuture directionTransition to OpenAFSTransition to Kerberos 5AFS home directories ?LSF/Grid integration ?Security issues ?Linux ?StabilityBackupsGrid IntegrationSatisfying DOE cyber security requirementsIntegration of Grid authentication and authorization with site authentication and authorizationStateful grid computing,