2023學年完整公開課版CloudComputing

CloudComputingNetworkComputingNetworkiscomputer(client-server)SeparationofFunctionalitiesEvolutionofComputingwithNetwork(1/2)2ClusterComputingTightlycoupledcomputingresources: CPU,storage,data,etc.UsuallyconnectedwithinaLANManagedasasingleresourceCommodity,OpensourceEvolutionofComputingwithNetwork(2/2)

GridComputingResourcesharingacrossseveraldomainsDecentralized,openstandardsGlobalresourcesharingUtilityComputingDon’tbuycomputers,leasecomputingpowerUpload,run,downloadOwnershipmodel3TheNextStep:CloudComputing

Serviceanddataareinthecloud,accessiblewithanydeviceconnectedtothecloudwithabrowserAkeytechnicalissuefordeveloper:ScalabilityServicesarenotknowngeographically45ApplicationsontheWebApplicationsontheWeb6ApplicationsontheWebCloudComputing

DefinitionCloudcomputingisaconceptofusingtheinternettoallowpeopletoaccesstechnology-enabledservices. Itallowsuserstoconsumeserviceswithoutknowledgeofcontroloverthetechnologyinfrastructurethatsupportsthem. -Wikipedia7MajorTypesofCloud

ComputeandDataCloudAmazonElasticComputingCloud(EC2),GoogleMapReduce,SciencecloudsProvideplatformforrunningsciencecodeHostCloudGoogleAppEngineHighly-available,faulttolerance,robustnessforwebcapability8CloudComputingExample-AmazonEC2/ec29CloudComputingExample-GoogleAppEngine

GoogleAppEngineAPIPythonruntimeenvironmentDatastoreAPIImagesAPIMailAPIMemcacheAPIURLFetchAPIUsersAPIAfreeaccountcanuseupto500MBstorage,enoughCPUandbandwidthforabout5millionpageviewsamonth/appengine/10CloudComputing

AdvantagesSeparationofinfrastructuremaintenancedutiesfromapplicationdevelopmentSeparationofapplicationcodefromphysicalresourcesAbilitytouseexternalassetstohandlepeakloadsAbilitytoscaletomeetuserdemandsquicklySharingcapabilityamongalargepoolofusers,improvingoverallutilization11CloudComputingSummary

CloudcomputingisakindofnetworkserviceandisatrendforfuturecomputingScalabilitymattersincloudcomputingtechnologyUsersfocusonapplicationdevelopmentServicesarenotknowngeographically12Countingthenumbersvs.Programmingmodel

PersonalComputerOnetoOneClient/ServerOnetoManyCloudComputingManytoMany13WhatPowersCloudComputinginGoogle?

CommodityHardwarePerformance:singlemachinenotinterestingReliabilityMostreliablehardwarewillstillfail:fault-tolerantsoftwareneededFault-tolerantsoftwareenablesuseofcommoditycomponentsStandardization:usestandardizedmachinestorunallkindsofapplications14WhatPowersCloudComputinginGoogle?InfrastructureSoftwareDistributedstorage:DistributedFileSystem(GFS)Distributedsemi-structureddatasystemBigTableDistributeddataprocessingsystemMapReduce15Whatisthecommonissuesofallthesesoftware?GoogleFileSystem

Filesbrokenintochunks(typically4MB)Chunksreplicatedacrossthreemachinesforsafety(tunable)Datatransfershappendirectlybetweenclientsandchunkservers16GFSUsage@Google

200+clustersFilesystemclustersofupto5000+machinesPoolsof10000+clients5+PetabyteFilesystemsAllinthepresenceoffrequentHWfailure17BigTable

Datamodel(row,column,timestamp)cellcontents18BigTable

Distributedmulti-levelsparsemapFault-tolerance,persistentScalableThousandofserversTerabytesofin-memorydataPetabytesofdisk-baseddataSelf-managingServerscanbeadded/removeddynamicallyServersadjusttoloadimbalance19WhynotjustusecommercialDB?

ScaleistoolargeorcostistoohighformostcommercialdatabasesLow-levelstorageoptimizationshelpperformancesignificantlyMuchhardertodowhenrunningontopofadatabaselayerAlsofunandchallengingtobuildlarge-scalesystems20BigTableSummary

DatamodelapplicabletobroadrangeofclientsActivelydeployedinmanyofGoogle’sservicesSystemprovideshigh-performancestoragesystemonalargescaleSelf-managingThousandsofserversMillionsofops/secondMultipleGB/sreading/writingCurrently–500+BigTablecellsLargestbigtablecellmanages–3PBofdataspreadoverseveralthousandmachines21DistributedDataProcessing

Problem:Howtocountwordsinthetextfiles?Inputfiles:NtextfilesSize:multiplephysicaldisksProcessingphase1:launchMprocessesInput:N/MtextfilesOutput:partialresultsofeachword’scountProcessingphase2:mergeMoutputfilesofstep122PseudoCodeofWordCount23TaskManagementLogisticsDecidewhichcomputerstorunphase1,makesurethefilesareaccessible(NFS-likeorcopy)Similarforphase2Execution:Launchthephase1programswithappropriatecommandlineflags,re-launchfailedtasksuntilphase1isdoneSimilarforphase2Automation:buildtaskscriptsontopofexistingbatchsystem24Technicalissues

Filemanagement:wheretostorefiles?StoreallfilesonthesamefileserverBottleneckDistributedfilesystem:opportunitytorunlocallyGranularity:howtodecideNandM?Joballocation:assignwhichtasktowhichnode?Preferlocaljob:knowledgeoffilesystemFault-recovery:whatifanodecrashes?RedundancyofdataCrash-detectionandjobre-allocationnecessary25MapReduceAsimpleprogrammingmodelthatappliestomanydata-intensivecomputingproblemsHidemessydetailsinMapReduceruntimelibraryAutomaticparallelizationLoadbalancingNetworkanddisktransferoptimizationHandleofmachinefailuresRobustnessEasytouse26MapReduceProgrammingModel

Borrowedfromfunctionalprogramming map(f,[x1,…,xm,…])=[f(x1),…,f(xm),…] reduce(f,x1,[x2,x3,…])=reduce(f,f(x1,x2),[x3,…])=…(continueuntilthelistisexhausted)Usersimplementtwofunctions map(in_key,in_value)(key,value)list reduce(key,[value1,…,valuem])f_value27MapReduce–ANewModelandSystemTwophasesofdataprocessingMap:(in_key,in_value){(keyj,valuej)|j=1…k}Reduce:(key,[value1,…valuem])(key,f_value)28MapReduceVersionofPseudoCode

29NoFileI/OOnlydataprocessinglogicExample–WordCount(1/2)InputisfileswithonedocumentperrecordSpecifyamapfunctionthattakesakey/valuepairkey=documentURLValue=documentcontentsOutputofmapfunctioniskey/valuepairs.Inourcase,output(w,”1”)onceperwordinthedocument30Example–WordCount(2/2)

MapReducelibrarygatherstogetherallpairswiththesamekey(shuffle/sort)Thereducefunctioncombinesthevaluesforakey.Inourcase,computethesumOutputofreducepairedwithkeyandsaved

31MapReduceFrameworkForcertainclassesofproblems,theMapReduceframeworkprovides:Automatic&efficientparallelization/distributionI/Oscheduling:RunmapperclosetoinputdataFault-tolerance:restartfailedmapperorreducertasksonthesameordifferentnodesRobustness:tolerateevenmassivefailures: e.g.large-scalenetworkmaintenance:oncelost1800outof2000machinesStatus/monitoring32TaskGranularityAndPipelining

Finegranularitytasks:manymoremaptasksthanmachinesMinimizestimeforfaultrecoveryCanpipelineshufflingwithmapexecutionBetterdynamicloadbalancingOftenuse200,000map/5000reducetaskswith2000machines33343536373839404142MapReduce:UsesatGoogle

Typicalconfiguration:200,000mappers,500reducerson2,000nodesBroadapplicabilityhasbeenapleasantsurpriseQualityexperiences,loganalysis,machinetranslation,ad-hocdataprocessingProductionindexingsystem:rewrittenwithMapReduce~10MapReductions,muchsimplerthanoldcode43MapReduceSummary

MapReduceisproventobeusefulabstractionGreatlysimplifieslarge-scalecomputationatGoogleFuntouse:focusonproblem,letlibrarydealwithmessydetails44ADataPlayground

MapReduce+BigTable+GFS=DataplaygroundSubstantialfractionofinternetavailableforprocessingEasy-to-useteraflops/petabytes,quickturn-aroundCoolproblems,greatcolleagues4546OpenSourceCloudSoftware:ProjectHadoop

GooglepublishedpapersonGFS(‘03),MapReduce(‘04)andBigTable(‘06)ProjectHadoopAnopensourceprojectwiththeApacheSoftwareFountationImplementGoogle’sCloudtechnologiesinJavaHDFS(GFS)andHadoopMapReduceareavailable.Hbase(BigTable)isbeingdevelopedGoogleisnotdirectlyinvolvedinthedevelopmentavoidconflictofinterest47IndustrialInterestinHadoop

Yahoo!hiredcoreHadoopdevelopersAnnouncedthattheirWebmapisproducedonaHadoopclusterwith2000hosts(dual/quadcores)onFeb.19,2008.AmazonEC2(ElasticComputeCloud)supportsHadoopWriteyourmapperandreducer,uploadyourdataandprogram,runandpaybyresourceutilizationTiff-to-PDFconversionof11million