用于高性能計算的 Linux 集群

October25|Slide1TimSkirvinandJimPhillipsTheoreticalandComputationalBiophysics,BeckmanInstituteLinuxClustersforHigh-PerformanceComputingJimPhillipsandTimSkirvinTheoreticalandComputationalBiophysicsBeckmanInstituteOctober25|Slide2TimSkirvinandJimPhillipsTheoreticalandComputationalBiophysics,BeckmanInstituteHPCvsHigh-AvailabilityTherearetwomajortypesofLinuxclusters:High-PerformanceComputingMultiplecomputersrunningasinglejobforincreasedperformanceHigh-AvailabilityMultiplecomputersrunningthesamejobforincreasedreliabilityWewillbetalkingabouttheformer!October25|Slide3TimSkirvinandJimPhillipsTheoreticalandComputationalBiophysics,BeckmanInstituteWhyClusters?Cheapalternativeto“bigiron”Localdevelopmentplatformfor“bigiron”codeBuilttotask(buyonlywhatyouneed)BuiltfromCOTScomponentsRunsCOTSsoftware(Linux/MPI)LoweryearlymaintenancecostsSinglefailuredoesnottakedownentirefacilityRe-deployasdesktopsor“throwaway”O(jiān)ctober25|Slide4TimSkirvinandJimPhillipsTheoreticalandComputationalBiophysics,BeckmanInstituteWhyNotClusters?Non-parallelizableortightlycoupledapplicationCostofportinglargeexistingcodebasetoohighNosourcecodeforapplicationNolocalexpertise(don’tknowUnix)NovendorhandholdingMassiveI/OormemoryrequirementsOctober25|Slide5TimSkirvinandJimPhillipsTheoreticalandComputationalBiophysics,BeckmanInstituteKnowYourUsersWhoareyoubuildingtheclusterfor?Yourselfandtwogradstudents?Yourselfandtwentygradstudents?Yourentiredepartmentoruniversity?Aretheyclueless,competitive,ormalicious?Howwillyoutoallocateresourcesamongthem?Willtheyexpectanexistinginfrastructure?Howwellwilltheytoleratesystemdowntimes?October25|Slide6TimSkirvinandJimPhillipsTheoreticalandComputationalBiophysics,BeckmanInstituteYourUsers’GoalsDoyouwantincreasedthroughput?Largenumberofqueuedserialjobs.Standardapplications,nochangesneeded.Ordecreasedturnaroundtime?Smallnumberofhighlyparalleljobs.Parallelizedapplications,changesrequired.October25|Slide7TimSkirvinandJimPhillipsTheoreticalandComputationalBiophysics,BeckmanInstituteYourApplicationThebestbenchmarkformakingdecisionsisyourapplicationrunningyourdataset.Designingaclusterisabouttrade-offs.Yourapplicationdeterminesyourchoices.Nosupercomputerrunseverythingwelleither.Neverbuyhardwareuntiltheapplicationisparallelized,ported,tested,anddebugged.October25|Slide8TimSkirvinandJimPhillipsTheoreticalandComputationalBiophysics,BeckmanInstituteYourApplication:ParallelPerformanceHowmuchmemorypernode?Howwoulditscaleonanidealmachine?Howisscalingaffectedby:Latency(timeneededforsmallmessages)?Bandwidth(timeperbyteforlargemessages)?Multiprocessornodes?Howfastdoyouneedtorun?October25|Slide9TimSkirvinandJimPhillipsTheoreticalandComputationalBiophysics,BeckmanInstituteBudgetFigureouthowmuchmoneyyouhavetospend.Don’tspendmoneyonproblemsyouwon’thave.Designthesystemtojustrunyourapplication.Neversolveproblemsyoucan’taffordtohave.Fastnetworkon20nodesorsloweron100?Don’tbuythehardwareuntil…Theapplicationisported,tested,anddebugged.Thescienceisreadytorun.October25|Slide10TimSkirvinandJimPhillipsTheoreticalandComputationalBiophysics,BeckmanInstituteEnvironmentTheclusterneedssomewheretolive.Youwon’twantitinyouroffice.Noteveninyourgradstudent’soffice.Clusterneeds:Space(keepthefiremartialhappy).PowerCoolingOctober25|Slide11TimSkirvinandJimPhillipsTheoreticalandComputationalBiophysics,BeckmanInstituteEnvironment:PowerMakesureyouhaveenoughpower.Kill-A-Watt$30atThinkGeek1.3GhzAthlondraws183VAatfullloadNewersystemsdrawmore;measureforyourself!MoreefficientpowersupplieshelpWallcircuitstypicallysupplyabout20AmpsAround12PCs@183VAmax(8-10forsafety)October25|Slide12TimSkirvinandJimPhillipsTheoreticalandComputationalBiophysics,BeckmanInstituteEnvironment:PowerFactorAthlon1333(Idle)1.25A98W137VAPF0.71Athlon1333(load)1.67A139W183VAPF0.76DualAthlonMP2600+2.89A246W319VAPF0.77DualXeon2.8GHz2.44A266W270VAPF0.985Moreefficientpowersuppliesdohelp!Alwaystestyourpowerunderload.W=VxAxPFOctober25|Slide13TimSkirvinandJimPhillipsTheoreticalandComputationalBiophysics,BeckmanInstituteEnvironment:UninterruptiblePower5kVAUPS($3,000)Holds24PCs@183VA(safely)WillneedtoworkoutbuildingpowertothemMaynotneedUPSforallsystems,justrootnodeOctober25|Slide14TimSkirvinandJimPhillipsTheoreticalandComputationalBiophysics,BeckmanInstituteEnvironment:CoolingBuildingACwillonlygetyousofarMakesureyouhaveenoughcooling.OnePC@183VAputsout~600BTUofheat.1tonofAC=12,000BTUs=~3500WattsCanrun~20CPUspertonofACOctober25|Slide15TimSkirvinandJimPhillipsTheoreticalandComputationalBiophysics,BeckmanInstituteHardwareManyimportantdecisionstomakeKeepapplicationperformance,users,environment,localexpertise,andbudgetinmindAnexerciseinsystemsintegration,makingmanyseparatecomponentsworkwellasaunitAreliablebutslightlyslowerclusterisbetterthanafastbutnon-functioningclusterAlwaysbenchmarkademosystemfirst!October25|Slide16TimSkirvinandJimPhillipsTheoreticalandComputationalBiophysics,BeckmanInstituteHardware:NetworkingTwomainoptions:GigabitEthernet–cheap($100-200/node),universallysupportedandtested,cheapcommodityswitchesupto48ports.24-portswitchesseemthebestbang-for-buckSpecialinterconnects:Myrinet–veryexpensive($thousandspernode),verylowlatency,logarithmiccostmodelforverylargeclusters.Infiniband–similar,lesscommon,notaswellsupported.October25|Slide17TimSkirvinandJimPhillipsTheoreticalandComputationalBiophysics,BeckmanInstituteHardware:OtherComponentsFilteredPower(Isobar,DataShield,etc)NetworkCables:buygoodones,you’llsavedebuggingtimelaterIfacableisatallquestionable,throwitaway!PowerCablesMonitorVideo/KeyboardCablesOctober25|Slide18TimSkirvinandJimPhillipsTheoreticalandComputationalBiophysics,BeckmanInstituteUserRulesofThumb1-4users:Yes,youstillwantaqueueingsystem.Planaheadtoavoididletimeandconflicts.5-20users:Putonepersoninchargeofrunningthings.Workoutafair-shareorreservationsystem.>20users:Userdocumentationandexamplesareessential.Decidewhomakesresourceallocationdecisions.October25|Slide19TimSkirvinandJimPhillipsTheoreticalandComputationalBiophysics,BeckmanInstituteApplicationRulesofThumb1-2programs:Don’tpayforanythingyouwon’tuse.Benchmark,benchmark,benchmark!Besuretouseyourtypicaldata.Trydifferentcompilersandcompileroptions.>2programs:SelectthemoststandardOSenvironment.Benchmarkthosethatwillrunthemost.Consideraspecializedclusterfordominantappsonly.October25|Slide20TimSkirvinandJimPhillipsTheoreticalandComputationalBiophysics,BeckmanInstituteParallelizationRulesofThumbThroughputiseasy…apprunsasis.Turnaroundisnot:Parallelspeedupislimitedby:Timespentinnon-parallelcode.Timespentwaitingfordatafromthenetwork.Improveserialperformancefirst:Profiletofindmosttime-consumingfunctions.Trynewalgorithms,libraries,handtuning.October25|Slide21TimSkirvinandJimPhillipsTheoreticalandComputationalBiophysics,BeckmanInstituteSomeDetailsMatterMoreWhatlimitingfactordoyouhitfirst?Budget?Space,power,andcooling?Networkspeed?Memoryspeed?Processorspeed?Expertise?October25|Slide22TimSkirvinandJimPhillipsTheoreticalandComputationalBiophysics,BeckmanInstituteLimitedbyBudgetDon’twastemoneysolvingproblemsyoucan’taffordtohaverightnow:RegularPCsonshelves(rollingcarts)GigabitnetworkingandmultiplejobsBenchmarkperformanceperdollar.Thelastdollaryouspendshouldbeonwhateverimprovesyourperformance.Askforequipmentfundsinproposals!October25|Slide23TimSkirvinandJimPhillipsTheoreticalandComputationalBiophysics,BeckmanInstituteLimitedbySpaceBenchmarkperformanceperrackConsiderallcombinationsof:RackmountnodesMoreexpensivebutnoperformancelossDual-processornodesLessmemorybandwidthperprocessorDual-coreprocessorsLessmemorybandwidthpercoreOctober25|Slide24TimSkirvinandJimPhillipsTheoreticalandComputationalBiophysics,BeckmanInstituteLimitedbyPower/CoolingBenchmarkperformanceperWattConsider:OpteronorPowerPCratherthanXeonDual-processornodesDual-coreprocessorsOctober25|Slide25TimSkirvinandJimPhillipsTheoreticalandComputationalBiophysics,BeckmanInstituteLimitedbyNetworkSpeedBenchmarkyourcodeatNCSA.10,000CPU-hoursiseasytoget.Tryrunningoneprocesspernode.Ifthatworks,buysingle-processornodes.TryMyrinet.Ifthatworks,canyourunatNCSA?Canyourunmore,smallerjobs?October25|Slide26TimSkirvinandJimPhillipsTheoreticalandComputationalBiophysics,BeckmanInstituteLimitedbySerialPerformanceIsitmemoryperformance?Try:Single-coreOpteronsSingle-processornodesLargercacheCPUsLowerclockspeedCPUsIsitreallytheprocessoritself?Try:HigherclockspeedCPUsDual-coreCPUsOctober25|Slide27TimSkirvinandJimPhillipsTheoreticalandComputationalBiophysics,BeckmanInstituteLimitedbyExpertiseThereisnosubstituteforalocalexpert.Qualifications:ComfortablewiththeUnixcommandline.ComfortablewithLinuxadministration.Clusterexperienceifyoucangetit.October25|Slide28TimSkirvinandJimPhillipsTheoreticalandComputationalBiophysics,BeckmanInstituteSystemSoftware“Linux”isjustastartingpoint.Operatingsystem,Libraries-messagepassing,numericalCompilersQueuingSystemsPerformanceStabilitySystemsecurityExistinginfrastructureconsiderationsOctober25|Slide29TimSkirvinandJimPhillipsTheoreticalandComputationalBiophysics,BeckmanInstituteScyldBeowulf/ClustermaticSinglefront-endmasternode:FullyoperationalnormalLinuxinstallation.Bprocpatchesincorporateslavenodes.Severelyrestrictedslavenodes:Minimuminstallation,downloadedatboot.Nodaemons,users,logins,scripts,etc.NoaccesstoNFSserversexceptformaster.HighlysecureslavenodesasaresultOctober25|Slide30TimSkirvinandJimPhillipsTheoreticalandComputationalBiophysics,BeckmanInstituteOscar/ROCKSEachnodeisafullLinuxinstallOffersaccesstoafilesystem.Softwaretoolshelpmanagetheselargenumbersofmachines.Stillmorecomplicatedthanonlymaintainingone“master”node.Bettersuitedforrunningmultiplejobsonasinglecluster,vsonejobonthewholecluster.October25|Slide31TimSkirvinandJimPhillipsTheoreticalandComputationalBiophysics,BeckmanInstituteSystemSoftware:CompilersNopointinbuyingfasthardwarejusttorunpoorperformingexecutablesGoodcompilersmightprovide50-150%performanceimprovementMaybecheapertobuya$2,500compilerlicensethantobuymorecomputenodesBenchmarkrealapplicationwithcompiler,getanevalcompilerlicenseifnecessaryOctober25|Slide32TimSkirvinandJimPhillipsTheoreticalandComputationalBiophysics,BeckmanInstituteSystemSoftware:MessagePassingLibrariesUsuallydictatedbyapplicationcodeChoosesomethingthatwillworkwellwithhardware,OS,andapplicationUser-spacemessagepassing?MPI:industrystandard,manyimplementationsbymanyvendors,aswellasseveralfreeimplementationsOthers:Charm++,BIP,FastMessagesOctober25|Slide33TimSkirvinandJimPhillipsTheoreticalandComputationalBiophysics,BeckmanInstituteSystemSoftware:NumericalLibrariesCanprovideahugeperformanceboostover“NumericalRecipes”orin-houseroutinesTypicallyhand-optimizedforeachplatformWhenapplicationsspendalargefractionofruntimeinlibrarycode,itpaystobuyalicenseforahighlytunedlibraryExamples:BLAS,FFTW,IntervallibrariesOctober25|Slide34TimSkirvinandJimPhillipsTheoreticalandComputationalBiophysics,BeckmanInstituteSystemSoftware:BatchQueueingClusters,althoughcheaperthan“bigiron”arestillexpensive,soshouldbeefficientlyutilizedTheuseofabatchqueueingsystemcankeepaclusterrunningjobs24/7Thingstoconsider:Allocationofsub-clusters?1-CPUjobsonSMPnodes?Examples:SunGridEngine,PBS,LoadLevelerOctober25|Slide35TimSkirvinandJimPhillipsTheoreticalandComputationalBiophysics,BeckmanInstituteSystemSoftware:OperatingSystemAnyannoyingmanagementorreliabilityissuesgethugelymultipliedinaclusterenvironment.PlanforsecurityfromtheoutsetClustershavespecialneeds;usesomethingappropriatefortheapplicationandhardwareOctober25|Slide36TimSkirvinandJimPhillipsTheoreticalandComputationalBiophysics,BeckmanInstituteSystemSoftware:InstallItYourselfDon’tusethevendor’spre-loadedOS.Theywouldlovetosellyou100licenses.Whathappenswhenyouhavetoreinstall?Doyouliketalkingtotechsupport?Arethoseflashygraphicsreallyuseful?Howmanysecurityholesarethere?October25|Slide37TimSkirvinandJimPhillipsTheoreticalandComputationalBiophysics,BeckmanInstituteSecurityTipsRestrictphysicalaccesstothecluster,ifpossible.Makesureyou’reinvolvedinalltours,tomakesurenobodytouchesanything.Ifyou’reoncampus,putyourclustersintotheFullyClosednetworkgroupMightcausesomelimitationsifyou’retryingtosubmitfromoff-siteWillcauseproblemswithGLOBUSThebuilt-infirewallisyourfriend!October25|Slide38TimSkirvinandJimPhillipsTheoreticalandComputationalBiophysics,BeckmanInstitutePurchasingTips:BeforeYouBeginGetyourbudgetWorkoutthespace,power,andcoolingcapacitiesoftheroom.StarttalkingtovendorsearlyButdon’tcommit!Don’tfallinlovewithanyonevendoruntilyou’velookedatthemall.October25|Slide39TimSkirvinandJimPhillipsTheoreticalandComputationalBiophysics,BeckmanInstitutePurchasingTips:DesignNotesMakesuretoordersomesparenodesSerialnodesandhot-swapsparesKeepthemrunningtomakesuretheywork.Ifpossible,installHDsonlyinheadnodeStatelawandUIUCpolicyrequiresallharddrivestobewipedbeforedisposalItdoesn’tmatterifthedriveneverstoredanything!Eachdrivewilltake8-10hourstowipe.Saveyourselfaworldofpaininafewyears……orjustgiveyourmachinestosomeothercampusgroup,andmakethemworryaboutit.October25|Slide40TimSkirvinandJimPhillipsTheoreticalandComputationalBiophysics,BeckmanInstitutePurchasingTips:GetLocalServiceIfanodedies,doyouwanttoshipit?Twochoices:Localbusiness(ChampaignComputer)Majorvendor(Sun)Askothersaboutresponsiveness.Designyourclustersothatyoucanstillrunjobsifacoupleofnodesaredown.October25|Slide41TimSkirvinandJimPhillipsTheoreticalandComputationalBiophysics,BeckmanInstitutePurchasingTips:DealingwithPurchasingYouwillwanttoputtheclusterorderonaPurchaseOrder(PO)Donotpayfortheclusteruntilitentirelyworks.Prepareaten-pointletterNecessaryforallpurchases>$25k.Examplesareavailablewithyourbusinessoffice(orbugusforourexamples).Thesearen’tdifficulttowrite,butwillprobablybenecessary.October25|Slide42TimSkirvinandJimPhillipsTheoreticalandComputationalBiophysics,BeckmanInstitutePurchasingTips:TheBidProcessAnypurchase>$28kmustgoupforbidException:sole-sourcevendorsNumbergrowseveryyearAddsamonthorsotothepurchasetimeIfyoucankeepthenumbersbelowthemagic$28k,doit!Thebidlimitmaybeleverageforvendorstodroptheirpricesjustbelowthelimit;planaccordingly.YouwillgetlotsofjunkbidsBeveryspecificaboutyourrequirementstokeepthemaway!October25|Slide43TimSkirvinandJimPhillipsTheoreticalandComputationalBiophysics,BeckmanInstitutePurchasingTips:WorkingtheBidProcessUsesole-sourcevendorswherepossible.ThisisamajorreasonwhywebuyfromSun.Checkwithyourpurchasingpeople.Thiswon’thelpyougetaroundthemonthtimeloss,astheitemstillhastobeposted.PurchaseyourclustersinsmallchunksOnlyworksifyou’relookingatarelativelysmallcluster.Again,youmaybeabletousethisasleveragewithyourvendortolowertheirprices.October25|Slide44TimSkirvinandJimPhillipsTheoreticalandComputationalBiophysics,BeckmanInstitutePurchasingTips:ReceivingYourEquipmentLetReceivingknowthatthemachinesarecoming.Itwilltakeupalotofspaceontheloadingdock.Workingwiththemtosavespacewillearnyougoodwill(andfasterturnaround).TakeyourmachinesoutofReceiving’sspaceassoonasreasonablypossible.October25|Slide45TimSkirvinandJimPhillipsTheoreticalandComputationalBiophysics,BeckmanInstitutePurchasingTips:ConsolidatedInventoryTrytoconvinceyourInventoryworkerstotageachcluster,andnoteachmachineIt’sreallygoingtoberunningasaclusteranyway(right?).Thiswillmakelifeeasieronyou.Repairsareeasierwhenyoudon’thavetoworryaboutinventorystickersThiswillmakelifeeasierforthem.3itemstotrackinsteadof72October25|Slide46TimSkirvinandJimPhillipsTheoreticalandComputationalBiophysics,BeckmanInstitutePurchasingTips:AssemblyGetextrahelpforassemblyIt’sreasonablyfunwork…aslongastheassemblylinegoesfast.Demandpizza.TesttheassemblyinstructionsbeforeyoubeginNothingismoreannoyingthanhavingtorealignalloftherailsafterthey’reallscrewedin.October25|Slide47TimSkirvinandJimPhillipsTheoreticalandComputationalBiophysics,BeckmanInstitutePurchasingTips:TestingandBenchmarkingTesttheclusterbeforeyouputitintoproduction!Samplejobs+cpuburnLookatpowerconsumptionTestfordeadnodesRemember:vendorsmakemistakes!Eventheirdemoapplicationsmaynotwork;checkforyourself.October25|Slide48TimSkirvinandJimPhillipsTheoreticalandComputationalBiophysics,BeckmanInstituteCaseStudiesThebestwaytoillustrateclusterdesignistolookathowsomebodyelsehasdoneit.TheTCBGrouphasdesignedfourseparateLinuxclustersinthelastsixyearsOctober25|Slide49TimSkirvinandJimPhillipsTheoreticalandComputationalBiophysics,BeckmanInstitute2001CaseStudyUsers:ManyresearcherswithMDsimulationsNeedtosupplementtimeonsupercomputersApplication:Notmemory-bound,runswellonIA32Scalesto32CPUswith100MbpsEthernetScalesto100+CPUswithMyrinetOctober25|Slide50TimSkirvinandJimPhillipsTheoreticalandComputationalBiophysics,BeckmanInstitute2001CaseStudy2Budget:Initially$20K,eventuallygrewto$100KEnvironment:Fullmachineroom,slowlyclearoutspaceUnder-utilized12kVAUPS,staffelectrician3tonchilledwaterairconditioner(Liebert)October25|Slide51TimSkirvinandJimPhillipsTheoreticalandComputationalBiophysics,BeckmanInstitute2001CaseStudy3Hardware:FastestAMDAthonCPUsavailable(1333MHz).FastCL2SDRAM,butnotDDR.Switched100MbpsEthernet,IntelEEProcards.Small40GBharddrivesandCD-ROMs.SystemSoftware:Scyldclustersof32machines,1job/cluster.ExistingDQS,NIS,NFS,etc.infrastructure.October25|Slide52TimSkirvinandJimPhillipsTheoreticalandComputationalBiophysics,BeckmanInstitute2003CaseStudyWhatchangedsince2001:50%increaseinprocessorspeed50%increaseinNAMDserialperformanceImprovedstabilityofSMPLinuxkernelInexpensivegigabitcardsand24-portswitchesNearlyfullmachineroomandpowersupplyPopularityofcompactformfactorcasesEmphasisoninteractiveMDofsmallsystemsOctober25|Slide53TimSkirvinandJimPhillipsTheoreticalandComputationalBiophysics,BeckmanInstitute2003CaseStudy2Budget:Initially$65K,eventuallygrewto~$100KEnvironment:SamegeneralmachineroomenvironmentAdditionalmachineroomspaceisavailableinserverroomJustswitchedtousingrack-mountequipmentStillusingtheoldclusters;don’twanttogetridofthementirelyNeedtobemorespace-consciousOctober25|Slide54TimSkirvinandJimPhillipsTheoreticalandComputationalBiophysics,BeckmanInstitute2003CaseSudy3Option#1:Singleprocessor,smallformfactornodes.HyperthreadedPentium4processors.32bit33MHzgigabitnetworkcards.24portgigabitswitch(24-processorclusters).Problems:NoECCmemory.Limitednetworkperformance.Toosmallfornext-generationvideocards.October25|Slide55TimSkirvinandJimPhillipsTheoreticalandComputationalBiophysics,BeckmanInstitute2003CaseStudy4Finaldecision:DualAthlonMP2600+innormalcases.NoharddrivesorCD-ROMs.64bit66MHzgigabitnetworkcards.24portgigabitswitch(48-procclusters).ClustermaticOS,bootslavesoffoffloppy.Floppieshaveprovenveryunreliable,especiallywhenleftinthedrives.Benefits:Serverclasshardwarew/ECCmemory.Maximumprocessorcountforlargesimulations.Maximumnetworkbandwidthforsmallsimulations.October25|Slide56TimSkirvinandJimPhillipsTheoreticalandComputationalBiophysics,BeckmanInstitute2003CaseStudy5Athlonclustersfrom2001recycled:36nodesou