深度學習最佳GPU

AllrightsreservedtoRun:ai.

NopartofthiscontentmaybeusedwithoutexpresspermissionofRun:ai.

PAGE

4

www.run.ai

TheBestGPUforDeepLearning

CriticalConsiderationsforLarge-ScaleAI

Traditionally,thetrainingphaseofthedeeplearningpipelinetakesthelongesttoachieve.Thisisnotonlyatimeconsumingprocess,butanexpensiveone.Themostvaluablepartofadeeplearningpipelineisthehumanelement.Datascientistsoftenwaitforhoursordaysfortrainingtocomplete,whichhurtstheirproductivityandthetimetobringnewmodelstomarket.

Tosignificantlyreducetrainingtime,youcanusedeeplearningGPUs,whichenableyouto

performAIcomputingoperationsinparallel.WhenassessingGPUs,youneedtoconsidertheabilitytointerconnectmultipleGPUs,thesupportingsoftwareavailable,licensing,dataparallelism,GPUmemoryuseandperformance.

Inthisguide,youwilllearn:

TheimportanceofGPUsindeeplearning 3

HowtochoosethebestGPUfordeeplearning 3

UsingconsumerGPUsfordeeplearning 4

BestdeeplearningGPUsfordatacenters 5

DGXfordeeplearningatscale 6

AutomatedDeepLearningGPUManagementWithRun:ai 8

WhyAreGPUsImportantinDeepLearning?

GPUFactorstoConsider

ThesefactorsaffectthescalabilityandeaseofuseoftheGPUsyouchoose:

Thelongestandmostresourceintensivephaseofmostdeeplearningimplementationsisthe

trainingphase.Thisphasecanbeaccomplishedinareasonableamountoftimeformodelswithsmallernumbersofparametersbutasyournumberincreases,yourtrainingtimedoesaswell.Thishasadualcost;yourresourcesareoccupiedforlongerandyourteamisleftwaiting,wastingvaluabletime.

Graphicalprocessingunits(GPUs)canreducethesecosts,enablingyoutorunmodelswithmassivenumbersofparametersquicklyandefficiently.ThisisbecauseGPUsenableyoutoparallelizeyourtrainingtasks,distributingtasksoverclustersofprocessorsandperformingcomputeoperationssimultaneously.

GPUsarealsooptimizedtoperformtargettasks,finishingcomputationsfasterthannon-specializedhardware.TheseprocessorsenableyoutoprocessthesametasksfasterandfreeyourCPUsforothertasks.Thiseliminatesbottleneckscreatedbycomputelimitations.

HowtoChoosetheBestGPUforDeepLearning?

SelectingtheGPUsforyourimplementationhassignificantbudgetandperformanceimplications.YouneedtoselectGPUsthatcansupportyourproject

inthelongrunandhavetheabilitytoscalethroughintegrationandclustering.Forlarge-scaleprojects,thismeansselectingproduction-gradeordatacenterGPUs.

AbilitytoInterconnectGPUs

WhenchoosingaGPU,youneedtoconsiderwhichunitscanbeinterconnected.InterconnectingGPUsisdirectlytiedtothescalabilityofyourimplementationandtheabilitytousemulti-GPUanddistributedtrainingstrategies.Typically,consumerGPUs

donotsupportinterconnection(NVlinkforGPUinterconnectswithinaserver,andInfiniband/RoCEforlinkingGPUsacrossservers)andNVIDIAhasremovedinterconnectionsonGPUsbelowRTX2080.

SupportingSoftware

NVIDIAGPUsarethebestsupportedintermsofmachinelearninglibrariesandintegrationwithcommonframeworks,suchasPyTorchorTensorFlow.TheNVIDIACUDAtoolkitincludesGPU-acceleratedlibraries,aCandC++compilerandruntime,andoptimizationanddebuggingtools.Itenablesyoutogetstartedrightawaywithoutworryingaboutbuildingcustomintegrations.

LearnmoreinourguidesaboutPyTorchGPUs,andNVIDIAdeeplearningGPUs.

Licensing

AnotherfactortoconsiderisNVIDIA’sguidanceregardingtheuseofcertainchipsindatacenters.Asofalicensingupdatein2018,theremayberestrictionsonuseofCUDAsoftwarewithconsumerGPUsin

adatacenter.Thismayrequireorganizationstotransitiontoproduction-gradeGPUs.

AlgorithmFactorsAffectiveGPUUse

Inourexperiencehelpingorganizationsoptimizelarge-scaledeeplearningworkloads,thefollowingarethethreekeyfactorsyoushouldconsiderwhenscalingupyouralgorithmacrossmultipleGPUs.

DataParallelism–Considerhowmuchdatayouralgorithmsneedtoprocess.Ifdatasetsaregoingtobelarge,investinGPUscapableofperformingmulti-GPUtrainingefficiently.Forverylargescaledatasets,makesurethatserverscancommunicatequicklywitheachotherandwithstoragecomponents,usingtechnologylikeInfiniband/RoCE,toenableefficientdistributedtraining.

MemoryUse–Areyougoingtodealwithlargedatainputstomodel?Forexample,modelsprocessingmedicalimagesorlongvideoshaveverylargetrainingsets,soyou’dwanttoinvestinGPUswithrelativelylargememory.Bycontrast,tabulardatasuchastextinputsforNLPmodelsaretypicallysmall,andyoucanmakedowithlessGPUmemory.

NVIDIATitanV

TheTitanVisaPCGPUthatwasdesignedforusebyscientistsandresearchers.ItisbasedonNVIDIA’s

VoltatechnologyandincludesTensorCores.TheTitanVcomesinStandardandCEOEditions.

TheStandardeditionprovides12GBmemory,110teraflopsperformance,a4.5MBL2cache,and3,072-bitmemorybus.TheCEOeditionprovides32GBmemoryand125teraflopsperformance,6MBcache,and4,096-bitmemorybus.Thelattereditionalsousesthesame8-HiHBM2memorystacksthatareusedinthe32GBTeslaunits.

PerformanceoftheGPU–Considerifyou’regoingtouseGPUsfordebugginganddevelopment.Inthiscaseyouwon’tneedthemostpowerfulGPUs.Fortuningmodelsinlongruns,youneedstrongGPUstoacceleratetrainingtime,toavoidwaitinghoursordaysformodelstorun.

UsingConsumerGPUsforDeepLearning

WhileconsumerGPUsarenotsuitableforlarge-scaledeeplearningprojects,theseprocessorscanprovideagoodentrypointfordeeplearning.

ConsumerGPUscanalsobeacheapersupplementforlesscomplextasks,suchasmodelplanningorlow-leveltesting.However,asyouscaleup,you’llwanttoconsiderdatacentergradeGPUsand

high-enddeeplearningsystemslikeNVIDIA’sDGXseries(learnmoreinthefollowingsections).

Inparticular,theTitanVhasbeenshowntoprovideperformancesimilartodatacenter-gradeGPUswhenitcomestoWordRNNs.Additionally,itsperformanceforCNNsisonlyslightlybelowhighertieroptions.TheTitanRTXandRTX2080Tiaren’tfarbehind.

NVIDIATitanRTX

TheTitanRTXisaPCGPUbasedonNVIDIA’sTuringGPUarchitecturethatisdesignedforcreativeandmachinelearningworkloads.ItincludesTensorCoreandRTCoretechnologiestoenableraytracingandacceleratedAI.

EachTitanRTXprovides130teraflops,24GBGDDR6memory,6MBcache,and11GigaRayspersecond.

Thisisdueto72TuringRTCoresand576multiprecisionTuringTensorCores.

NVIDIAGeForceRTX2080Ti

TheGeForceRTX2080TiisaPCGPUdesignedforenthusiasts.ItisbasedontheTU102graphics

processor.EachGeForceRTX2080Tiprovides11GBofmemory,a352-bitmemorybus,a6MBcache,androughly120teraflopsofperformance.

BestDeepLearningGPUsforLarge-ScaleProjectsandDataCenters

ThefollowingareGPUsrecommendedforuseinlarge-scaleAIprojects.

NVIDIATeslaA100

TheA100isaGPUwithTensorCoresthatincorporatesmulti-instanceGPU(MIG)technology.Itwasdesignedformachinelearning,dataanalytics,andHPC.TheTeslaA100ismeanttobescaledtouptothousandsofunitsandcanbepartitioned

intosevenGPUinstancesforanysizeworkload.EachTeslaA100providesupto624teraflopsperformance,40GBmemory,1,555GBmemorybandwidth,and600GB/sinterconnects.

NVIDIATeslaV100

TheNVIDIATeslaV100isaTensorCoreenabledGPUthatwasdesignedformachinelearning,deeplearning,andhighperformancecomputing(HPC).ItispoweredbyNVIDIAVoltatechnology,whichsupportstensorcoretechnology,specializedforacceleratingcommontensoroperationsindeeplearning.EachTeslaV100provides149teraflopsofperformance,upto32GBmemory,anda4,096-bitmemorybus.

NVIDIATeslaP100

TheTeslaP100isaGPUbasedonanNVIDIAPascalarchitecturethatisdesignedformachinelearningandHPC.EachP100providesupto21teraflopsofperformance,16GBofmemory,anda4,096-bitmemorybus.

NVIDIATeslaK80

TheTeslaK80isaGPUbasedontheNVIDIAKeplerarchitecturethatisdesignedtoacceleratescientificcomputinganddataanalytics.Itincludes4,992NVIDIACUDAcoresandGPUBoost?technology.EachK80providesupto8.73teraflopsofperformance,24GBofGDDR5memory,and480GBofmemorybandwidth.

GoogleTPU

SlightlydifferentareGoogle’stensorprocessingunits(TPUs).TPUsarechiporcloud-based,application-specificintegratedcircuits(ASIC)fordeeplearning.TheseunitsarespecificallydesignedforusewithTensorFlowandareavailableonlyonGoogleCloudPlatform.

EachTPUcanprovideupto420teraflopsofperformanceand128GBhighbandwidthmemory(HBM).Therearealsopodversionsavailablethatcanprovideover100petaflopsofperformance,32TBHBM,anda2Dtoroidalmeshnetwork.

DGXforDeepLearningatScale

CPUO

CPUO

NIC

NIC

NIC

NIC

PCIeSwitches

PCIeSwitches

V100GPU3

V100GPU0

V100GPU4

V100GPU7

NVLink

V100GPU2

V100GPU1

V100GPU5

V100GPU6

PCIe

QPI

TheNVIDIADGXsystemsarefullstacksolutionsdesignedforenterprise-grademachinelearning.ThesesystemsarebasedonasoftwarestackthatisoptimizedforAI,multi-nodescalability,andenterprise-gradesupport.

YoucanimplementtheDGXstackincontainersoronbaremetal.Thistechnologyismeanttobe

plug-and-playandisfullyintegratedwithNVIDIAdeeplearninglibrariesandsoftwaresolutions.DGXisavailableforserver-classworkstations,servers,orpods.Below,theserveroptionsareintroduced.

DGX-1

TheDGX-1isaGPUserverbasedontheUbuntuLinuxHostOS.ItintegrateswithRedHatsolutionsandincludestheDIGITSdeeplearningtrainingapplication,theNVIDIADeepLearningSDK,theCUDAtoolkit,andtheDockerEngineUtilityforNVIDIAGPU.

EachDGX-1provides:

TwoIntelXeonCPUsfordeeplearningframeworkcoordination,boot,andstoragemanagement

Upto8TeslaV100TensorCoresGPUswith32GBofmemory

300Gb/sNVLinkinterconnects

800GB/scommunicationwithlow-latency

Single480GBbootOSSSDandfour1.92TBSASSSDs(7.6TBtotal)configuredasaRAID0stripedvolume

DGX-2

TheDGX-2isthenextlevelupfromtheDGX-1.ItisbasedontheNVSwitchnetworkingfabricforgreaterparallelismandscalability.

EachDGX-2provides:

DGXA100

TheDGXA100isdesignedtobeauniversalsystemformachinelearningworkloads,includinganalytics,training,andinference.ItisfullyoptimizedforCUDA-X.TheDGXA100canbestackedwithotherA100unitstocreatemassiveAIclusters,includingtheNVIDIADGXSuperPOD.

Twopetaflopsofperformance

2X960GBNVMESSDsforOSstorageand30TBofSSDstorage

EachDGXA100provides:

Fivepetaflopsofperformance

16TeslaV100TensorCoreGPUswith32GBofmemory

1.6TB/slow-latency,bi-directionalbandwidth

1.5TBsystemmemory

EightA100TensorCoreGPUswith40GBmemory

SixNVSwitchesfor4.8TBbi-directionalbandwidth

Two64-coreAMDCPUsfordeeplearningframeworkcoordination,boot,andstorage

TwoXeonPlatinumCPUsfordeeplearningframeworkcoordination,boot,andstorage

1TBsystemmemory,2x1.92TBM.2NVMEdrivesforOSstorageand15TBSSDstorage

TwohighI/Oethernetcards

AllrightsreservedtoRun:ai.

NopartofthiscontentmaybeusedwithoutexpresspermissionofRun:ai.

PAGE

8

www.run.ai

AutomatedDeepLearningGPUManagementWithRun:ai

Run:aiautomatesresourcemanagementandworkloadorchestrationformachinelearninginfrastructure.WithRun:ai,youcanautomaticallyrunasmanycomputeintensiveexperimentsasneeded.

HerearesomeofthecapabilitiesyougainwhenusingRun:ai:

MLOps

Intoday’shighlycompetitiveeconomy,enterprisesarelookingtoartificialintelligenceingeneralandmachineanddeeplearninginparticulartotransformbigdataintoactionableinsightsthatcanhelpthembetteraddresstheirtargetaudiences,improvetheirdecision-makingprocesses,andstreamlinetheirsupplychainsandproductionprocesses,tomentionjustafewofthemanyusecasesoutthere.InordertostayaheadofthecurveandcapturethefullvalueofML,however,companiesmuststrategicallyembraceMLOps.

SeetoparticlesinourMLOpsguide:

Advancedvisibility:createanefficientpipelineofresourcesharingbypoolingGPUcomputeresources

Nomorebottlenecks:youcansetupguaran