spark大數(shù)據(jù)分析平臺Spark快速計算講師

Spark13DATAGURU專業(yè)數(shù)據(jù)分析Spark大數(shù)據(jù)快速計算平風 課程詳 Spark大數(shù)據(jù)快速計算平風

DATAGURU專業(yè)數(shù)據(jù)分析RDD的編SparkSpark大數(shù)據(jù)快速計算平風

DATAGURU專業(yè)數(shù)據(jù)分析SparkSpark大數(shù)據(jù)快速計算平風

DATAGURU專業(yè)數(shù)據(jù)分析Driver

Ajarcontainingtheuser'sSparkapplication.Insomecasesuserswillwanttocreatean"uberjar"containingtheirapplicationalongwithitsdependencies.Theuser'sjarshouldneverincludeHadooporSparklibraries,however,thesewillbeaddedatTheprocessrunningthemain()functionoftheapplicationandcreatingtheCluster Anexternalserviceforacquiringresourcesonthecluster(e.g.standalonemanager,Mesos,Deploy Distinguisheswherethedriverprocessruns.In"cluster"mode,theframeworklaunchesthedriverinsideofthecluster."client"mode,thesubmitterlaunchesthedriveroutsideoftheWorker Anynodethatcanrunapplicationcodeinthe Aprocesslaunchedforanapplicationonaworkernode,thatrunstasksandkeepsdatainmemoryordiskstoragethem.Eachapplicationhasitsown Aunitofworkthatwillbesenttoone AparallelcomputationconsistingofmultipletasksthatgetsspawnedinresponsetoaSparkaction(e.g.save,collect);you'llseethistermusedinthedriver'slogs.EachjobgetsdividedintosmallerEachjobgetsdividedintosmallersetsoftaskscalledstagesthatdependoneachother(similartothemapandreducestagesinMapReduce);you'llseethistermusedinthedriver'slogs.DATAGURU專業(yè)數(shù)據(jù)分析SparkRDDSpark大數(shù)據(jù)快速計算平風

DATAGURU專業(yè)數(shù)據(jù)分析Spark編程模Spark大數(shù)據(jù)快速計算平風

DATAGURU專業(yè)數(shù)據(jù)分析RDDSpark大數(shù)據(jù)快速計算平風

DATAGURU專業(yè)數(shù)據(jù)分析描述scala>valmap1=rdd.map(_*3)res0:Array[Int]=Array(3,6,9,12,scala>valrddscala>valfiltered=res2:Array[String]=Array(jifeng,scala>valrdd=sc.parallelize(List("aaaddfdc","dcisfun"))scala>valfm=rdd.flatMap(str=>str.split(""))res3:Array[String]=Array(aa,ad,df,dc,dc,is,Spark大數(shù)據(jù)快速計算平風

DATAGURU專業(yè)數(shù)據(jù)分析描述reduceByKey(func,scala>valwd=fm.map(word=>(word,1))scala>wdnum.collect()Array[(String,Int)]=Array((aa,1),(is,1),(df,1),(dc,2),(ad,1),Array[(String,Int)]=Array((aa,1),(ad,1),(dc,2),(df,1),(fun,1),scala>valcntWrd=wdnum.map{case(word,count)=>(count,word)}scala>cntWrd.groupByKey().collect()Array[(Int,Iterable[String])]=Array(( pactBuffer(dc)),( pactBuffer(aa,is,df,ad,scala>fm.distinct().collect()Array[String]=Array(aa,is,df,dc,ad,Spark大數(shù)據(jù)快速計算平風

DATAGURU專業(yè)數(shù)據(jù)分析描述 scala>rdd.reduce(_+_)Int=scala>valrdd=rdd:Array[String]=Array(A,B,C,scala>rdd.count()Long=scala>rdd.take(3)Array[String]=Array(A,B,Spark大數(shù)據(jù)快速計算平風

DATAGURU專業(yè)數(shù)據(jù)分析描述scala>rdd.first()String=AScala>valrdd=scala>$ls//查看part-00000part-00001Spark大數(shù)據(jù)快速計算平風

DATAGURU專業(yè)數(shù)據(jù)分析描述Spark大數(shù)據(jù)快速計算平風

DATAGURU專業(yè)數(shù)據(jù)分析haRDhaRDRDaquet文件、Jh查詢he注意Spark1.1使用registerTempTable代替1.0版本的Spark1.1在hiveContext中，hql()將被棄用，sql()將代替hql()來提交查詢語句，統(tǒng)一了接口使用 表是一個臨時表，生命周期只在所定義的sqlContext或hiveContext實例之中spark1.1提供了語法解析器選項spark.sql.dialect，目前有兩種語法解析器，sql語法解析器和hiveql語法解析器sqlContext現(xiàn)在只支持sql語法解析器（SQL-92語法）hiveContext現(xiàn)在支持sql語法解析器和hivesql語法解析器，默認為hivesql語法解析器，用戶可以通過配置切換成sql語法解析器Spark大數(shù)據(jù)快速計算平風

DATAGURU專業(yè)數(shù)據(jù)分析 [jifeng@feng03~]$mkdirhive[jifeng@feng03~]$cdhive/[jifeng@feng02hive]$tarzxfapache-hive-0.13.1-bin.tar.gz[jifeng@feng02hive]$lsSpark大數(shù)據(jù)快速計算平風

DATAGURU專業(yè)數(shù)據(jù)分析Hive[jifeng@feng03hive]$viexport2修改 下的文[jifeng@feng03conf]$perties.template[jifeng@feng02conf]$mvhive-env.sh.templatehive-env.sh[jifeng@feng02conf]$cphive-default.xml.templatehive-Spark大數(shù)據(jù)快速計算平風

DATAGURU專業(yè)數(shù)據(jù)分析Hive修改 下的文export[jifeng@feng02conf]$cd[jifeng@feng02bin]$vihive-#Defaulttouseexport"hive-config.sh"73L,2023CSpark大數(shù)據(jù)快速計算平風

DATAGURU專業(yè)數(shù)據(jù)分析Hive配置[jifeng@feng03conf]$vihive-<property><description>JDBCconnectstringforaJDBC <description>DriverclassnameforaJDBCSpark大數(shù)據(jù)快速計算平風

DATAGURU專業(yè)數(shù)據(jù)分析Hive<description>usernametouseagainstmetastore<description>passwordtouseagainstmetastoreSpark大數(shù)據(jù)快速計算平風

DATAGURU專業(yè)數(shù)據(jù)分析Hive啟動[jifeng@feng02hadoop-2.6.0]$./sbin/start-啟動[jifeng@feng03hive]$15/09/0516:31:10WARNconf.HiveConf:DEPRECATED:hive.metastore.ds.retry.*nolongerhasanyeffect.Usehive.hmshandler.retry.*insteadLogginginitializedusingconfigurationinjar:file:/home/jifeng/hive/apache-hive-0.13. hive>createdatabasesale;Timetaken:1.188Spark大數(shù)據(jù)快速計算平風

DATAGURU專業(yè)數(shù)據(jù)分析hive>createtableshop(locationid namestring)rowformatdelimitedfieldsterminatedbyexceptionwasthrownwhileadding/validatingclass(es):Specifiedkeywastoolong;maxkeylengthis767atatjava.lang.reflect.Constructor.newInstance(Constructor.java:526)atcom.mysql.jdbc.Util.getInstance(Util.java:360)mysql>alterdatabasehivecharactersetlatin1;QueryOK,1rowaffected(0.00sec)Spark大數(shù)據(jù)快速計算平風

DATAGURU專業(yè)數(shù)據(jù)分析hive>createtableshop(locationidint,name namestring)rowformatdelimitedfieldsterminatedby',';Timetaken:0.884hive>createtablesell(locationidint,fyearint,amountdouble,numdouble)rowformatdelimitedfieldsterminatedbyTimetaken:0.158hive>showtables;Timetaken:0.105seconds,Fetched:2row(s)Spark大數(shù)據(jù)快速計算平風

DATAGURU專業(yè)數(shù)據(jù)分析上傳文件到jifeng@feng03hive]$hadoopfsputhome/jifeng/code/db/shop.txtuser/jifeng/fire/hive>loaddatainpath'/user/jifeng/fire/shop.txt'intotableshop;Loadingdatatotabledefault.shopTimetaken:0.657hive>loaddatainpath'/user/jifeng/fire/sell.txt'intotablesell;Loadingdatatotabledefault.sellTimetaken:0.38Spark大數(shù)據(jù)快速計算平風

DATAGURU專業(yè)數(shù)據(jù)分析Hivesupportisenabledbyaddingthe-Phiveand-Phive-thriftserverflagstoSpark’sConfigurationofHiveisdonebyplacingyourhive-site.xmlfileinspark-sql>select*fromshopwhere15/09/0516:54:52INFOscheduler.TaskSchedulerImpl:Addingtaskset2.0with2tasks15/09/0516:54:52INFOstorage.BlockManagerInfo:Addedbroadcast_3_piece0inmemoryon10:58340(size:35.2KB,free:267.2MB)15/09/0516:54:52INFOscheduler.TaskSetManager:Finishedtask0.0instage2.0(TID3)in574mson10(1/2)15/09/0516:54:53INFOscheduler.DAGScheduler:ResultStage2(processCmdatCliDriver.java:423)finishedin0.617s15/09/0516:54:53INFOscheduler.DAGScheduler:Job2finished:processCmdatCliDriver.java:423,took0.680182s Spark大數(shù)據(jù)快速計算平風

DATAGURU專業(yè)數(shù)據(jù)分析 importimportorg.apache.spark.sql.hive.HiveContextimportorg.apache.spark.mllib.linalg.VectorsvalsqlContext=newvalsqldata=sqlContext.sql("selectlocationid,sum(num)allnum,sum(amount)allamountfromsellgroupbylocationid")Spark大數(shù)據(jù)快速計算平風

DATAGURU專業(yè)數(shù)據(jù)分析valparsedData=sqldata.mapcaseRow(_,allnum,allamount)} valnumClusters=3valnumIterations=valmodel=KMeans.train(parsedData,numClusters,Spark大數(shù)據(jù)快速計算平風

DATAGURU專業(yè)數(shù)據(jù)分析valresult1=sqldata.map{caseRow(locationid,allnum,allamount)vallinevectore=Vectors.dense(features)valprediction=locationid+""+allnum+""+allamoun