Hive編程入門-weibo-大數(shù)據(jù)文檔資料

集團技術(shù)平臺部-核心系統(tǒng)-海量數(shù)據(jù)天照1–OpenSourceMapReduceframework–支持SQL語義的大規(guī)模數(shù)據(jù)分析工具–離線/數(shù)據(jù)倉庫應(yīng)用?為超大數(shù)據(jù)集設(shè)計的計算/擴展能力?統(tǒng)一的元數(shù)據(jù)管理整體架構(gòu)?Client端應(yīng)用程序?元數(shù)據(jù)?編程接口?環(huán)境變量–HIVE_HOME–HIVE_CONF_DIR–HADOOP_HOME/HADOOP_CONF_DIR–hive-default.xml（默認(rèn)不做修改）/hive-site.xml（用戶的配置）–perties?選擇一個元數(shù)據(jù)服務(wù)器–int/bigint/smallint/tinyint–boolean–double/float–string?沒有精度/長度設(shè)定(col_namedata_type,...)等同于：…–INSERTOVERWRITET–INSERTOVERWRITETABLE[WHEREwhere_condition]–join_table:table_referenceJOINtable_factor[{LEFT|RIGHT|FULL}OUTER|LEFTSEMI]JOINtable_referencejoin_condition–table_factor:tbl_name[alias]|table–join_condition:ONequality_expression(ANDequality_expression)*equality_expression:-e'quotedquerystring'Sqlfromcommandline$HIVE_HOME/bin/hive-e'selecta.colfromtab1a‘$HIVE_HOME/bin/hive-e'selecta.colfromtab1a'-hiveconfmapred.reduce.tasks=32$HIVE_HOME/bin/hive-f/home/my/hive-script.sql$HIVE_HOME/bin/hive-i/home/my/hive-init.sqlquitorexittocomeoutofinteractiveshell.Usethistosetvalueofparticularconfigurationvariable.Onethingtonotehereisthatifyoumisspellthevariablename,cliwillnotshowanerror.Thiswillgiveallpossiblehadoop/hiveconfigurationvariables.JARresourcesarealsoaddedtotheJavaclasspath.ThisisrequiredinordertoreferenceobjectstheycontainsuchasUDF's.executedfscommandcommandfromhiveshell?<querystring>executeshivequeryandprintsresultstostdoutexecutessqlfromfileandprintsresultstostdouthive>setmapred.reduce.tasks=32;//設(shè)置值hive>setmapred.reduce.tasks;//顯示值hive>selecta.*fromtab1;hive>addjar/home/tianzhao/hive/hive-1.1.2/lib/hive_contrib.jar;hive>CREATETEMPORARYFUNCTIONdate_formatAS'com.taobao.hive.udf.UDFDateFormat';hive>source/home/tianzhao.blj/test_others_hive/union.sql;顯示partition的信息?Java語言編寫的用戶自定義函數(shù)–UDF-1:1–UDAF-N:1–UDTF-1:N–CREATETEMPORARYFUNCTIONfunction_nameASclass_name–implementevaluate()function–extendsUDAF/GenericUDAF–implementUSING'my_map_script'(AScolName(','colName)*)?(clusterBy?|distribute)REDUCEUSING'my_reduce_script'(AS