Big-Data-大數據介紹(全英)

BigDataWeipingChenTopicsWhatisBigData?Why‘BigData’isabigdeal?NoSQLvsSQLHowtoDealwithBigData?What’sHadoop/MapReduce?RDBMSvsHadoop/MapReduceBigdataplayers/SoftwareTools/PlatformsExamplesWhatIsBigData?CapturingandmanaginglotsofinformationWorkingwithmanynewtypesofdataStructure/UnstructuredExploitingthesemassesofinformationandnewdatatypeswithnewstylesofapplicationsBiggerthanTerabytesvolume,variety,velocity,variabilityWhy‘BigData’isabigDealBigdatadiffersfromtraditionalinformationinmind-bendingways:

NotknowingwhybutonlywhatThechallengewithleadershipisthatit’sverydrivenbygutinstinctinmostcasesAirtravelerscannowfigureoutwhichflightsarelikeliesttobeontime,thankstodatascientistswhotrackedadecadeofflighthistorycorrelatedwithweatherpatternsPublishersusedatafromtextanalysisandsocialnetworkstogivereaderspersonalizednews.healthcareisoneofthebiggestopportunities,IfwehadelectronicrecordsofAmericansgoingbackgenerations,we'dknowmoreaboutgeneticpropensities,correlationsamongsymptoms,andhowtoindividualizetreatments.Googlemapsearchcorrelateto“Openretailstoreetc.”WhatThisMeansforYou

BigDatacanhelpacompanydomanythings:ProfilecustomersDeterminepricingstrategiesIdentifycompetitiveadvantagesBettertargetadvertisingInforminternalresearchandproductdevelopmentStrengthencustomerserviceMainstepsinadoptingananalyticalsystemWhatWillWeAnalyze?DoWeBuyorBuild?AreWeReadytoInvest?DoWeUnderstandtheImpact?ChallengesInformationgrowthProcessingpowerPhysicalstoragediskcapacityincreasedramatically100MB/Sreadfromdisk(bottleneck)dataseekingtimeisslowthandatatransferringDataissuesCostsRecentlyITTrendCommodityhardwareDistributedfilesystemsOpensourceoperatingsystems,databases,andotherinfrastructureSignificantlycheaperstorageService-orientedarchitecture

BigDataChainCollectDataIngest/CleanData(OriginallyETL.Existingschema)Humanexploration/Infrastructure/DataminingStore/ArchiveShare(decisionmake,othersystem)Measure/feedbackACIDACID(Atomicity,Consistency,Isolation,Durability)

(A)whenyoudosomethingtochangeadatabasethechangeshouldworkorfailasawhole(C)thedatabaseshouldremainconsistent(thisisaprettybroadtopic)(I)ifotherthingsaregoingonatthesametimetheyshouldn'tbeabletoseethingsmid-update(D)ifthesystemblowsup(hardwareorsoftware)thedatabaseneedstobeabletopickitselfbackup;andifitsaysitfinishedapplyinganupdate,itneedstobecertainMapReduceDividingandconqueringHighlyfaulttolerantnodesareexpectedtofail?Everydatablock(bydefault)replicatedon3nodes(isalsorackaware)DifficulttoimplementRDBMSfixed-schema,row-orienteddatabaseswithACIDpropertiesandasophisticatedSQLqueryengine.Theemphasisisonstrongconsistency,referentialintegrity,abstractionfromthephysicallayer,andcomplexqueriesthroughtheSQLlanguage.easilycreatesecondaryindexes,performcomplexinnerandouterjoins,count,sum,sort,group,andpageyourdataacrossanumberoftables,rows,andcolumns.RDBMSvsMapReduceRDBMSMapReducemostlystructureddataunstructureddatadatainternalstructurenone(doesinprocess)normalizedneednon-nomalizeNotes:1.relationaldatabasesstartincorporatingsomeoftheideasfromMapReduce(suchasAsterData’sandGreenplum’sdatabases)2.theotherdirection,ashigher-levelquerylanguagesbuiltonMapReduce(suchasPigandHive)makeMapReducesystemsmoreapproachablefortraditionaldatabaseprogrammers.ArchitechuresHowdoesMapReduceworkHDFS(HadoopDistributedFileSystem)

DataisstoredonlocaldiskandprocessingisdonelocallyonthecomputerwiththedataCanworkwithrawdatastoredinfilesystemordatabaseTwosteps:MapandReduce

MapMapReduceuseskey/valuepairs.(Traditionallyusingrowsandcolumns)

Example:lastname/chen

withdrawamount/20

transactiondate/06-23-2013Reducealltheintermediatevaluesforagivenoutputkeyarecombinedtogetherintoalist.Thereduce()functionthencombinestheintermediatevaluesintooneormorefinalvaluesforthesamekey.HadoopHadoopisdesignedtoabstractawaymuchofthecomplexityofdistributedprocessingDifferentfromGRIDcomputingWidelyusedSocialmedia(e.g.,Facebook,Twitter)

Lifesciences

Financialservices

Retail

GovernmentHadoopArchitectureApplicationlayer/enduseraccesslayera.JobTracker(workloadmanagementlayer)b.Distributedparallelfilesystems/datalayerHadoopImplementationHadoopisdesignedtorunjobsthatlastminutesorhoursontrusted,dedicatedhardwarerunninginasingledatacenterwithveryhighaggregatebandwidthinterconnectsDesignofHDFSNamenodes(TheMaster)Managemetadata/filetreesDatanodes(Workers)

store/retrievedatablockDatanodesdonotuseRAIDdisk.HDFSround-robinsHDFSblocksbetweenalldisks.RAIDlimitedbytheslowestdiskonthearray.

LimitationsofHDFSLow-latencydataaccessLotsofsmallfilesMultiplewriters,arbitraryfilemodificationsHDFSBlock64MB/128MB(normaldiskblock512KB).minimize‘seek’timefixedsizeratherthanfile,easystorage/replication%hadoopfsck/-files–blocks%hadoopfs–help(regularfilesystemoperation)%hadoopfs-copyFromLocalinput/docs/quangle.txthdfs://localhost/user/tom/quangle.txt%hadoopfs-mkdirbooks%hadoopfs-lsDataflowsFormatandTypesMapReducemodelindetail,and,inparticular,howdatainvariousformats,fromsimpletexttostructuredbinaryobjects,canbeusedwiththismodelmap:(K1,V1)→list(K2,V2)reduce:(K2,list(V2))→list(K3,V3)TextfileOnthetopoftheCrumpettyTreeTheQuangleWanglesat,Buthisfaceyoucouldnotsee,OnaccountofhisBeaverHat.isdividedintoonesplitoffourrecords.Therecordsareinterpretedasthefollowingkey-valuepairs:(0,OnthetopoftheCrumpettyTree)

(33,TheQuangleWanglesat,)(57,Buthisfaceyoucouldnotsee,)(89,OnaccountofhisBeaverHat.)DataFileMapreduceSpecialFeatureCounterSortingJoinsShuffle

MapReduceguaranteesthattheinputtoeveryreducerissortedbykey.Theprocessbywhichthesystemperformsthesort—andtransfersthemapoutputstothereducersasinputs-ShuffleInstallHadoop%cd/usr/local%sudotarxzfhadoop-x.y.z.tar.gzchangetheowneroftheHadoopfilestobethehadoopuserandgroup:%sudochown-Rhadoop:hadoophadoop-x.y.zLayers/Players--continueExtract,transform,load(ETL)

IBMInfoSphereDataStageInformaticaPervasiveTalendDatawarehouse

Oracle,Teradata,IBMNetezza,Greenplum

PIG–HelpHadoopPigisascriptinglanguageforexploringlargedatasetsAPigLatinprogramismadeupofaseriesofoperations,ortransformations,thatareappliedtotheinputdatatoproduceoutput2.PigexecutionenvironmenttranslatesintoanexecutablerepresentationandthenrunsHbaseHBaseisadistributedcolumn(family)-orienteddatabasebuiltontopofHDFS.HBaseistheHadoopapplicationtousewhenyourequirereal-timeread/writerandom-accesstoverylargedatasetsHBasetablesarelikethoseinanRDBMS,onlycellsareversioned,rowsaresorted,andcolumnscanbeaddedontheflybytheclientaslongasthecolumnfamilytheybelongtopreexists.Hbase--continueRegions

Eachregioncomprisesasubsetofatable’srowsprovidewaystoreadorwriteindividualrecordsefficientlybasedonHadoopHiveHive—anopensourcedatawarehousingandSQLinfrastructurebuiltontopofHadoopCloudera’sDistributionforHadoopCloudera’sDistributionforHadoopisbasedonthemostrecentstableversionofApacheHadoopwithnumerouspatches,backports,andupdatesEvaluateCriteriaHighscalabilityLowlatencyPredictabilityHighavailabilityEasymanagementMulti-tenancyBigDataRealtimeProcessingGoogleBigQueryisawebservicethatletsyoudointeractiveanalysisofmassivedatasets—uptobillionsofrowsTwitter’sStormClouderaImpalaNoSQLNoSQLreferstodocument-orienteddatabasesSQLdoesn’tscalewellhorizontally(addmoreserverswhichCloudisgoodat)Itisschemaless.Butnotformless(JSONformat).JSON:datainterchangeformatMongoDatabaseCouchDatabaseNoSQLBaseModelBaseModelBasicAvailability:spreaddataacrossmanystoragesystemswithahighdegreeofreplicationSoftState:dataconsistencyisthedeveloper'sproblemandshouldnotbehandledbythedatabase.EventualConsistency:atsomepointinthefuture,datawillconvergetoaconsistentstate.Noguaranteesaremade“when”JSONStructure{field1:value1,field2:value2…fieldN:valueN}varmydoc={_id:ObjectId("5099803df3f4948bd2f98391"),name:{first:"Alan",last:"Turing"},birth:newDate('Jun23,1912'),death:newDate('Jun07,1954'),contribs:["Turingmachine","Turingtest",…],views:NumberLong(1250000)}RDBMSvsNoSQLXszcRowDB:001:10,Smith,Joe,40000;002:12,Jones,Mary,50000;003:11,Johnson,Cathy,44000;004:22,Jones,Bob,55000;index:001:40000;002:50000;003:44000;004:55000;ColumnDB:10:001,12:002,11:003,22:004;Smith:001,Jones:002,Johnson:003,Jones:004;Joe:001,Mary:002,Cathy:003,Bob:004;40000:001,50000…;Smith:001,Jones:002,004,Johnson:003;…BenefitsColumn-orientedorganizationsaremoreefficientwhenanaggregateneedstobecomputedovermanyrowsbutonlyforanotablysmallersubsetofallcolumnsofdata,becausereadingthatsmallersubsetofdatacanbefasterthanreadingalldata.Column-orientedorganizationsaremoreefficientwhennewvaluesofacolumnaresuppliedforallrowsatonce,becausethatcolumndatacanbewrittenefficientlyandreplaceoldcolumndatawithouttouchinganyothercolumnsfortherows.Row-orientedorganizationsaremoreefficientwhenmanycolumnsofasinglerowarerequiredatthesametime,andwhenrow-sizeisrelativelysmall,astheentirerowcanberetrievedwithasinglediskseek.Row-orientedorganizationsaremoreefficientwhenwritinganewrowifallofthecolumndataissuppliedatthesametime,astheentirerowcanbewrittenwithasinglediskseek.SQLvsNonSQLAgoodcompromiseistodesignyoursystemwith3logicalDBs1.NormalSQLDBusedbyyouradminapplicationtocreatecontent.

2.No-SQLDBforfront-end/public/high-volumeapplicaitonusedbythepublicinternet.

3.ThelastDBisforanalyticalreportingsystemusingcubesandallthatgoodstuff.ThendataflowsfromtheAdminDBtotheclientNo-SQLDBwhensomeone"Publishes"apieceofcontent,theclient(NoSQL)dbprovidesveryfastreadaccessandrecordsuserinteractionswiththecontent.ThenyouhaveascheduledjobthatpullsthedatafromtheclientDBintothereportingsystem.SinceAdmin,client,andreportingareoftenseparateapps,eachapplicationteamcanworkwithdataintheformatthatbestservestheapplicationandthetransitionfromonesystemtotheotherishandledintheservicelayers.BigDataSolutionsCloudera:ClouderaEnterpriseMicrosoft:WindowsAzureHDInsightServiceGoogle:BigQueryAmazon:DynamoDBIBM:InfoSphereStreams/NetezzaEMC:GreenplumTeraData:AsterMapReducePlatformOracle:Hadoop/MapreduceBigDataconnectorsBigDataProjectFailReasonsLackofcooperationamongdepartmentsLackof

staff

experiencedinBigDataSecurityPoorplanningRealExamplesofBigDataProjectsConsumerproductcompaniesandretailorganizationsaremonitoringsocialmedialikeFacebookandTwittertogetanunprecedentedviewintocustomerbehavior,preferences,andproductperception.Manufacturersaremonitoringminutevibrationdatafromtheirequipment,whichchangesslightlyasitwearsdown,topredicttheoptimaltimetoreplaceormaintain.Replacingittoosoonwastesmoney;replacingittoolatetriggersanexpensiveworkstoppageManufacturersarealsomonitoringsocialnetworks,butwithadifferentgoalthanmarketers:Theyareusingittodetectaftermarketsupportissuesbeforeawarrantyfailurebecomespubliclydetrimental.FinancialServicesorganizationsareusingdataminedfromcustomerinteractionstosliceanddicetheirusersintofinelytunedsegments.Thisenablesthesefinancialinstitutionstocreateincreasinglyrelevantandsophisticatedoffers.ContinuationAdvertisingandmarketingagenciesaretrackingsocialmediatounderstandresponsivenesstocampaigns,promotions,andotheradvertisingmediums.InsurancecompaniesareusingBigDataanalysistoseewhichhomeinsuranceapplicationscanbeimmediatelyprocessed,andwhichonesneedavalidatingin-personvisitfromanagent.Byembracingsocialmedia,retailorganizationsareengagingbrandadvocates,changingtheperceptionofbrandantagonists,andevenenablingenthusiasticcustomerstoselltheirproducts.Hospitalsareanalyzingmedicaldataandpatientrecordstopredictthosepatientsthatarelikelytoseekreadmissionwithinafewmonthsofdischarge.Thehospitalcantheninterveneinhopesofpreventinganothercostlyhospitalstay.Web-basedbusinessesaredevelopinginformationproductsthatcombinedatagatheredfromcustomerstooffermoreappealingrecommendationsandmoresuccessfulcouponprograms.Thegovernmentismakingdatapublicatboththenational,state,andcitylevelforuserstodevelopnewapplicationsthatcangeneratepublicgood.Sportsteamsareusingdatafortrackingticketsalesandevenfortrackingteamstrategies.StartingBigDataProjectsNYTD(NationalYouthinTransitionDatabase)DocumentationSearchDynamicSQLtableWWWlogfilesHealthCare:extractingnames,locations,dates,products,diseases,Rx,conditions,etc.,fromtextNYTD(NationalYouthTransitinalDatabase)DatacolectionsystemtotracktheStatesaretocollectinformationoneachyouthwhoreceivesindependentlivingservicespaidfororprovidedbytheStateagencythatadministerstheCFCIP.Second,StatesaretocollectdemographicandoutcomeinformationoncertainyouthinfostercarewhomtheStatewillfollowovertimetocollectadditionaloutcomeinformationthe