大數據介紹英文方案

BIGDATAEVERYMINUTE…1,388

cabs2,777privatecarsDidirideshailed:EVERYMINUTE…395,833PeopleloginToWeChat194,444peoplearevideooraudiochattingEVERYMINUTE…625,000YoukuTudouvideosbeingwatchedEVERYMINUTE…64,814postsandrepostsonWeiboSEARCH4,166,667

searchqueriesEVERYMINUTE…774

peoplebuysomethingonAlibaba’smarketplacesUS$1,133,942spentonAlibaba1Definition2Characteristic3NoSQL4RDBMS5MapReduceCONTENTS6Applications1Definition1DefinitionBIGDATAvolumeofdataimportantdataonaday-to-daybasisforbetterdecisions2Characteristic2CharacteristicVolumeThequantityofgeneratedandstoreddata.VarietyThetypeandnatureofthedata.Thequalityofcaptureddatacanvarygreatly,affectingaccurateanalysis.VelocityInthiscontext,thespeedatwhichthedataisgeneratedandprocessedtomeetthedemandsandchallengesthatlieinthepathofgrowthanddevelopment.VariabilityInconsistencyofthedatasetcanhamperprocessestohandleandmanageit.Veracity3NoSQL3NoSQLNoSQLreferstodocument-orienteddatabasesSQLdoesn’tscalewellhorizontally.Itisschemaless.Butnotformless(JSONformat).JSON:datainterchangeformatMongoDatabaseCouchDatabase3NoSQLBasicAvailabilityspreaddataacrossmanystoragesystemswithahighdegreeofreplication.SoftStateEventualConsistencyBaseModeldataconsistencyisthedeveloper'sproblemandshouldnotbehandledbythedatabase.atsomepointinthefuture,datawillconvergetoaconsistentstate.Noguaranteesaremade“when”.3NoSQL{field1:value1,field2:value2…fieldN:valueN}varmydoc={_id:ObjectId("5099803df3f4948bd2f98391"),name:{first:"Alan",last:"Turing"},birth:newDate('Jun23,1912'),death:newDate('Jun07,1954'),contribs:["Turingmachine","Turingtest",…],views:NumberLong(1250000)}JSONStructure3NoSQLRDBMSvsNoSQLXszcRowDB:001:10,Smith,Joe,40000;002:12,Jones,Mary,50000;003:11,Johnson,Cathy,44000;004:22,Jones,Bob,55000;index:001:40000;002:50000;003:44000;004:55000;ColumnDB:10:001,12:002,11:003,22:004;Smith:001,Jones:002,Johnson:003,Jones:004;Joe:001,Mary:002,Cathy:003,Bob:004;40000:001,50000…;Smith:001,Jones:002,004,Johnson:003;…3NoSQLBenefitsColumn-orientedorganizationsaremoreefficientwhenanaggregateneedstobecomputedovermanyrowsbutonlyforanotablysmallersubsetofallcolumnsofdata,becausereadingthatsmallersubsetofdatacanbefasterthanreadingalldata.Column-orientedorganizationsaremoreefficientwhennewvaluesofacolumnaresuppliedforallrowsatonce,becausethatcolumndatacanbewrittenefficientlyandreplaceoldcolumndatawithouttouchinganyothercolumnsfortherows.Row-orientedorganizationsaremoreefficientwhenmanycolumnsofasinglerowarerequiredatthesametime,andwhenrow-sizeisrelativelysmall,astheentirerowcanberetrievedwithasinglediskseek.Row-orientedorganizationsaremoreefficientwhenwritinganewrowifallofthecolumndataissuppliedatthesametime,astheentirerowcanbewrittenwithasinglediskseek.3NoSQLSQLvsNonSQLAgoodcompromiseistodesignyoursystemwith3logicalDBs1.NormalSQLDBusedbyyouradminapplicationtocreatecontent.2.No-SQLDBforfront-end/public/high-volumeapplicaitonusedbythepublicinternet.3.ThelastDBisforanalyticalreportingsystemusingcubesandallthatgoodstuff.ThendataflowsfromtheAdminDBtotheclientNo-SQLDBwhensomeone"Publishes"apieceofcontent,theclient(NoSQL)dbprovidesveryfastreadaccessandrecordsuserinteractionswiththecontent.ThenyouhaveascheduledjobthatpullsthedatafromtheclientDBintothereportingsystem.SinceAdmin,client,andreportingareoftenseparateapps,eachapplicationteamcanworkwithdataintheformatthatbestservestheapplicationandthetransitionfromonesystemtotheotherishandledintheservicelayers.4RDBMS4RDBMSfixed-schema,row-orienteddatabaseswithACIDpropertiesandasophisticatedSQLqueryengineTheemphasisisonstrongconsistency,referentialintegrity,abstractionfromthephysicallayer,andcomplexqueriesthroughtheSQLlanguage.easilycreatesecondaryindexes,performcomplexinnerandouterjoins,count,sum,sort,group,andpageyourdataacrossanumberoftables,rows,andcolumns.5MapReduceDividingandconqueringHighlyfaulttolerantEverydatablockreplicatedon3nodesDifficulttoimplement5MapReduce5ComparisonRDBMSMapReduceDatasizeGBPBAccessInteractiveandBatchBatchUpdatesRead/WritemanytimesWriteonce,ReadmanytimesStructureStaticSchemaDynamicSchemeIntegratedHigh(ACID)LowScalingNolinerLinerDBARatio1：401：30005HowdoesMapReduceworkMapReduceuseskey/valuepairs.(Traditionallyusingrowsandcolumns) --Mapalltheintermediatevaluesforagivenoutputkeyarecombinedtogetherintoalist. --ReduceThereducefunctionthencombinestheintermediatevaluesintooneormorefinalvaluesforthesamekey.--ReduceTwosteps:MapandReduce6Application6GovernmentTheuseandadoptionofbigdatawithingovernmentalprocessesisbeneficialandallowsefficienciesintermsofcost,productivity,andinnovation,butdoesnotcomewithoutitsflaws.Dataanalysisoftenrequiresmultiplepartsofgovernment(centralandlocal)toworkincollaborationandcreatenewandinnovativeprocessestodeliverthedesiredoutcome.Belowarethethought[bywhom?]leadingexampleswithinthegovernmentalbigdataspace.6HealthcareBigdataanalyticshashelpedhealthcareimprovebyprovidingpersonalizedmedicineandprescriptiveanalytics,clinicalriskinterventionandpredictiveanalytics,wasteandcarevariabilityreduction,automatedexternalandinternalreportingofpatientdata,standardizedmedicaltermsandpatientregistriesandfragmentedpointsolutions.6EducationAMcKinseyGlobalInstitutestudyfoundashortageof1.5millionhighlytraineddataprofessionalsandmanagersandanumberofuniversitiesincludingUniversityofTennesseeandUCBerkeley,havecreatedmastersprogramstomeetthisdemand.Privatebootcampshavealsodevelopedprogramstomeetthatdemand,includingfreeprogramslikeTheDataIncubatororpaidprogramslikeGeneralAssembly.6InternetofThingsBigDataandtheIoTworkinconjunction.Fromamediaperspective,dataisthekeyderivativeofdeviceinter-connectivityandallowsaccuratetargeting.TheInternetofThings,withthehelpofbigdata,thereforetransformsthemediaindustry,companiesandevengovernmen