大數(shù)據(jù)基礎課程設計報告

1、-. z.大數(shù)據(jù)根底課程設計報告一、工程簡介：使用hadoop中的hive、mapreduce以及HBASE對網(wǎng)上的一個搜狗五百萬的數(shù)進展了一個比擬實際的數(shù)據(jù)分析。搜狗五百萬數(shù)據(jù)，是經(jīng)過處理后的搜狗搜索引擎生產(chǎn)數(shù)據(jù)，具有真實性，大數(shù)據(jù)性，能夠較好的滿足分布式計算應用開發(fā)課程設計的數(shù)據(jù)要求。搜狗數(shù)據(jù)的數(shù)據(jù)格式為：訪問時間t 用戶 IDt查詢詞t 該 URL 在返回結果中的排名t 用戶點擊的順序號t 用戶點擊的 URL。其中，用戶 ID 是根據(jù)用戶使用瀏覽器訪問搜索引擎時的 Cookie 信息自動賦值，即同一次使用瀏覽器輸入的不同查詢對應同一個用戶 ID。二、操作要求1.將原始數(shù)據(jù)加載到HDFS平

2、臺。2.將原始數(shù)據(jù)中的時間字段拆分并拼接，添加年、月、日、小時字段。3.將處理后的數(shù)據(jù)加載到HDFS平臺。4.以下操作分別通過MR和Hive實現(xiàn)。查詢總條數(shù)非空查詢條數(shù)無重復總條數(shù)獨立UID總數(shù)查詢頻度排名頻度最高的前50詞查詢次數(shù)大于2次的用戶總數(shù)查詢次數(shù)大于2次的用戶占比Rank在10以的點擊次數(shù)占比直接輸入URL查詢的比例查詢搜索過仙劍奇?zhèn)b傳的uid，并且次數(shù)大于35.將4每步驟生成的結果保存到HDFS中。6.將5生成的文件通過Java API方式導入到HBase一表。7.通過HBase shell命令查詢6導出的結果。三、實驗流程將原始數(shù)據(jù)加載到HDFS平臺將原始數(shù)據(jù)中的時間字段拆分并

3、拼接，添加年、月、日、小時字段編寫1個腳本sogou-log-e*tend.sh，其中sogou-log-e*tend.sh的容為：#!/bin/bashinfile=$1outfile=$2awk -F t print $0tsubstr($1,0,4)年tsubstr($1,5,2)月tsubstr($1,7,2)日tsubstr($1,8,2)hour $infile $outfile處理腳本文件：結果為：將處理后的數(shù)據(jù)加載到HDFS平臺hadoop fs -put sogou.500w.utf8.e*t /以下操作分別通過MR和Hive實現(xiàn).hive實現(xiàn)1.查看數(shù)據(jù)庫：show dat

4、abases;2.創(chuàng)立數(shù)據(jù)庫: create database sogou;3.使用數(shù)據(jù)庫: use sogou;4.查看所有表：show tables;5.創(chuàng)立sougou表：Create table sogou(time string,uuid string,name string,num1 int,num2 int,url string) Row format delimited fields terminated by t;6.將本地數(shù)據(jù)導入到Hive表里：Load data local inpath /root/sogou.500w.utf8 into table sogou;7.查

5、看表信息：desc sogou;查詢總條數(shù)select count(*) from sogou;非空查詢條數(shù)select count(*) from sogou where name is not null and name !=;無重復總條數(shù)select count(*) from (select * from sogou group by time,num1,num2,uuid,name,url having count(*)=1) a;獨立UID總數(shù)select count(distinct uuid) from sogou;查詢頻度排名頻度最高的前50詞select name,coun

6、t(*) as pd from sogou group by name order by pd desc limit 50;6查詢次數(shù)大于2次的用戶總數(shù)select count(a.uuid) from (select uuid,count(*) ast from sogou group by uuid havingt 2) a;7查詢次數(shù)大于2次的用戶占比select count(*) from (select uuid,count(*) ast from sogou group by uuid havingt 2) a;Rank在10以的點擊次數(shù)占比select count(*) from

7、 sogou where num13;.MapReduce實現(xiàn)(import的各種包省略)查詢總條數(shù)public class MRCountAll public static Integer i = 0; public static boolean flag = true; public static class CountAllMap e*tends Mapper Override protected void map(Object key, Te*t value, Mapper.Conte*t conte*t) throws IOE*ception, InterruptedE*ceptio

8、n i+; public static void runcount(String Inputpath, String Outpath) Configuration conf = new Configuration(); conf.set(fs.defaultFS, hdfs:/0:9000); Job job = null; try job = Job.getInstance(conf, count); catch (IOE*ception e) / TODO Auto-generated catch block e.printStackTrace(); job.setJa

9、rByClass(MRCountAll.class); job.setMapperClass(CountAllMap.class); job.setOutputKeyClass(Te*t.class); job.setOutputValueClass(Te*t.class); try FileInputFormat.addInputPath(job, new Path(Inputpath); catch (IllegalArgumentE*ception e) / TODO Auto-generated catch block e.printStackTrace(); catch (IOE*c

10、eption e) / TODO Auto-generated catch block e.printStackTrace(); FileOutputFormat.setOutputPath(job, new Path(Outpath); try job.waitForpletion(true); catch (ClassNotFoundE*ception e) / TODO Auto-generated catch block e.printStackTrace(); catch (IOE*ception e) / TODO Auto-generated catch block e.prin

11、tStackTrace(); catch (InterruptedE*ception e) / TODO Auto-generated catch block e.printStackTrace(); public static void main(String args) throws E*ception runcount(/sogou/data/sogou.500w.utf8, /sogou/data/CountAll); System.out.println(總條數(shù): + i); 非空查詢條數(shù)public class CountNotNull public static String S

12、tr = ; public static int i = 0; public static boolean flag = true; public static class wyMap e*tends Mapper Override protected void map(Object key, Te*t value, Mapper.Conte*t conte*t) throws IOE*ception, InterruptedE*ception String values = value.toString().split(t); if (!values2.equals(null) & valu

13、es2 != ) conte*t.write(new Te*t(values1), new IntWritable(1); i+; public static void run(String inputPath, String outputPath) Configuration conf = new Configuration(); conf.set(fs.defaultFS, hdfs:/0:9000); Job job = null; try job = Job.getInstance(conf, countnotnull); catch (IOE*ception e)

14、 / TODO Auto-generated catch block e.printStackTrace(); assert job != null; job.setJarByClass(CountNotNull.class); job.setMapperClass(wyMap.class); job.setOutputKeyClass(Te*t.class); job.setOutputValueClass(IntWritable.class); try FileInputFormat.addInputPath(job, new Path(inputPath); catch (Illegal

15、ArgumentE*ception e) e.printStackTrace(); catch (IOE*ception e) e.printStackTrace(); try FileOutputFormat.setOutputPath(job, new Path(outputPath); job.waitForpletion(true); catch (ClassNotFoundE*ception e) e.printStackTrace(); catch (IOE*ception e) e.printStackTrace(); catch (InterruptedE*ception e)

16、 e.printStackTrace(); public static void main(String args) run(/sogou/data/sogou.500w.utf8, /sogou/data/CountNotNull); System.out.println(非空條數(shù): + i); 無重復總條數(shù)public class CountNotRepeat public static int i = 0; public static class NotRepeatMap e*tends Mapper Override protected void map(Object key, Te*

17、t value, Mapper.Conte*t conte*t) throws IOE*ception, InterruptedE*ception String te*t = value.toString(); String values = te*t.split(t); String time = values0; String uid = values1; String name = values2; String url = values5; conte*t.write(new Te*t(time+uid+name+url), new Te*t(1); public static cla

18、ss NotRepeatReduc e*tends Reducer Override protected void reduce(Te*t key, Iterable values, Reducer.Conte*t conte*t) throws IOE*ception, InterruptedE*ception i+; conte*t.write(new Te*t(key.toString(),new IntWritable(i); public static void main(String args) throws IOE*ception, ClassNotFoundE*ception,

19、 InterruptedE*ception Configuration conf = new Configuration(); conf.set(fs.defaultFS, hdfs:/0:9000); Job job = null; try job = Job.getInstance(conf, countnotnull); catch (IOE*ception e) / TODO Auto-generated catch block e.printStackTrace(); assert job != null; job.setJarByClass(CountNotRe

20、peat.class); job.setMapperClass(NotRepeatMap.class); job.setReducerClass(NotRepeatReduc.class); job.setOutputKeyClass(Te*t.class); job.setOutputValueClass(Te*t.class); try FileInputFormat.addInputPath(job, new Path(/sogou/data/sogou.500w.utf8); catch (IllegalArgumentE*ception e) e.printStackTrace();

21、 catch (IOE*ception e) e.printStackTrace(); try FileOutputFormat.setOutputPath(job, new Path(/sogou/data/CountNotRepeat); job.waitForpletion(true); catch (ClassNotFoundE*ception e) e.printStackTrace(); catch (IOE*ception e) e.printStackTrace(); catch (InterruptedE*ception e) e.printStackTrace(); Sys

22、tem.out.println(無重復總條數(shù)為: + i); 獨立UID總數(shù)public class CountNotMoreUid public static int i = 0; public static class UidMap e*tends Mapper Override protected void map(Object key, Te*t value, Mapper.Conte*t conte*t) throws IOE*ception, InterruptedE*ception String te*t = value.toString(); String values = t

23、e*t.split(t); String uid = values1; conte*t.write(new Te*t(uid), new Te*t(1); public static class UidReduc e*tends Reducer Override protected void reduce(Te*t key, Iterable values, Reducer.Conte*t conte*t) throws IOE*ception, InterruptedE*ception i+; conte*t.write(new Te*t(key.toString(),new IntWrit

24、able(i); public static void main(String args) throws IOE*ception, ClassNotFoundE*ception, InterruptedE*ception Configuration conf = new Configuration(); conf.set(fs.defaultFS, hdfs:/0:9000); Job job = null; try job = Job.getInstance(conf, countnotnull); catch (IOE*ception e) / TODO Auto-ge

25、nerated catch block e.printStackTrace(); assert job != null; job.setJarByClass(CountNotNull.class); job.setMapperClass(UidMap.class); job.setReducerClass(UidReduc.class); job.setOutputKeyClass(Te*t.class); job.setOutputValueClass(Te*t.class); try FileInputFormat.addInputPath(job, new Path(/sogou/dat

26、a/sogou.500w.utf8); catch (IllegalArgumentE*ception e) e.printStackTrace(); catch (IOE*ception e) e.printStackTrace(); try FileOutputFormat.setOutputPath(job, new Path(/sogou/data/CountNotMoreUid); job.waitForpletion(true); catch (ClassNotFoundE*ception e) e.printStackTrace(); catch (IOE*ception e)

27、e.printStackTrace(); catch (InterruptedE*ception e) e.printStackTrace(); System.out.println(獨立UID條數(shù): + i); 查詢頻度排名頻度最高的前50詞public class CountTop50 public static class TopMapper e*tends Mapper Te*t te*t =new Te*t(); Override protected void map(LongWritable key, Te*t value,Conte*t conte*t) throws IOE*c

28、eption, InterruptedE*ception String line= value.toString().split(t); String keys = line2; te*t.set(keys); conte*t.write(te*t,new LongWritable(1); public static class TopReducer e*tends Reducer Te*t te*t = new Te*t(); TreeMap map = new TreeMap(); Override protected void reduce(Te*t key, Iterable valu

29、e, Conte*t conte*t) throws IOE*ception, InterruptedE*ception int sum=0;/key出現(xiàn)次數(shù) for (LongWritable lte*t : value) sum+=lte*t.get(); map.put(sum,key.toString(); /去前50條數(shù)據(jù) if(map.size()50) map.remove(map.firstKey(); Override protected void cleanup(Conte*t conte*t) throws IOE*ception, InterruptedE*ceptio

30、n for(Integer count:map.keySet() conte*t.write(new Te*t(map.get(count), new LongWritable(count); public static void main(String args) throws IOE*ception, ClassNotFoundE*ception, InterruptedE*ception Configuration conf = new Configuration(); conf.set(fs.defaultFS, hdfs:/0:9000); Job job = J

31、ob.getInstance(conf, count); job.setJarByClass(CountTop50.class); job.setJobName(Five); job.setOutputKeyClass(Te*t.class); job.setOutputValueClass(LongWritable.class); job.setMapperClass(TopMapper.class); job.setReducerClass(TopReducer.class); FileInputFormat.addInputPath(job, new Path(/sogou/data/s

32、ogou.500w.utf8); FileOutputFormat.setOutputPath(job, new Path(/sogou/data/CountTop50); job.waitForpletion(true); 查詢次數(shù)大于2次的用戶總數(shù)public class CountQueriesGreater2 public static int total = 0; public static class MyMaper e*tends Mapper protected void map(Object key, Te*t value, Mapper.Conte*t conte*t) t

33、hrows IOE*ception, InterruptedE*ception String str = value.toString().split(t); Te*t word; IntWritable one = new IntWritable(1); word = new Te*t(str1); conte*t.write(word, one); public static class MyReducer e*tends Reducer Override protected void reduce(Te*t arg0, Iterable arg1, Reducer.Conte*t arg

34、2) throws IOE*ception, InterruptedE*ception / arg0是一個單詞 arg1是對應的次數(shù) int sum = 0; for (IntWritable i : arg1) sum += i.get(); if(sum2) total=total+1; /arg2.write(arg0, new IntWritable(sum); public static void main(String args) throws IOE*ception, ClassNotFoundE*ception, InterruptedE*ception Configurati

35、on conf = new Configuration(); conf.set(fs.defaultFS, hdfs:/0:9000); / 1.實例化一個Job Job job = Job.getInstance(conf, si*); / 2.設置mapper類 job.setMapperClass(MyMaper.class); / 3.設置biner類不是必須的 / job.setbinerClass(MyReducer.class); / 4.設置Reducer類 job.setReducerClass(MyReducer.class); / 5.設置輸出key的

36、數(shù)據(jù)類型 job.setOutputKeyClass(Te*t.class); / 6.設置輸出value的數(shù)據(jù)類型 job.setOutputValueClass(IntWritable.class); / 設置通過哪個類查找job的Jar包 job.setJarByClass(CountQueriesGreater2.class); / 7.設置輸入路徑 FileInputFormat.addInputPath(job, new Path(/sogou/data/sogou.500w.utf8); / 8.設置輸出路徑 FileOutputFormat.setOutputPath(job,

37、 new Path(/sogou/data/CountQueriesGreater2); / 9.執(zhí)行該作業(yè) job.waitForpletion(true); System.out.println(查詢次數(shù)大于2次的用戶總數(shù)： + total + 條); 查詢次數(shù)大于2次的用戶占比public class CountQueriesGreaterPro public static int total1 = 0; public static int total2 = 0; public static class MyMaper e*tends Mapper Override protected

38、void map(Object key, Te*t value, Mapper.Conte*t conte*t) throws IOE*ception, InterruptedE*ception total2+; String str = value.toString().split(t); Te*t word; IntWritable one = new IntWritable(1); word = new Te*t(str1); conte*t.write(word, one); / 執(zhí)行完畢后就是一個單詞對應一個value(1) public static class MyReducer

39、 e*tends Reducer Override protected void reduce(Te*t arg0, Iterable arg1, Reducer.Conte*t arg2) throws IOE*ception, InterruptedE*ception / arg0是一個單詞 arg1是對應的次數(shù) int sum = 0; for (IntWritable i : arg1) sum += i.get(); if(sum2) total1+; arg2.write(arg0, new IntWritable(sum); public static void main(Str

40、ing args) throws IOE*ception, ClassNotFoundE*ception, InterruptedE*ception System.out.println(seven begin); Configuration conf = new Configuration(); conf.set(fs.defaultFS, hdfs:/0:9000); / 1.實例化一個Job Job job = Job.getInstance(conf, seven); / 2.設置mapper類 job.setMapperClass(MyMaper.class);

41、/ 3.設置biner類不是必須的 / job.setbinerClass(MyReducer.class); / 4.設置Reducer類 job.setReducerClass(MyReducer.class); / 5.設置輸出key的數(shù)據(jù)類型 job.setOutputKeyClass(Te*t.class); / 6.設置輸出value的數(shù)據(jù)類型 job.setOutputValueClass(IntWritable.class); / 設置通過哪個類查找job的Jar包 job.setJarByClass(CountQueriesGreaterPro.class); / 7.設置輸

42、入路徑 FileInputFormat.addInputPath(job, new Path(/sogou/data/sogou.500w.utf8); / 8.設置輸出路徑 FileOutputFormat.setOutputPath(job, new Path(/sogou/data/CountQueriesGreaterPro); / 9.執(zhí)行該作業(yè) job.waitForpletion(true); System.out.println(total1=+total1+ttotal2=+total2); float percentage = (float)total1/(float)to

43、tal2; System.out.println(查詢次數(shù)大于2次的用戶占比為： + percentage*100+%); System.out.println(over); Rank在10以的點擊次數(shù)占比public class CountRank public static int sum1 = 0; public static int sum2 = 0; public static class MyMapper e*tends Mapper Override protected void map(Object key, Te*t value, Mapper.Conte*t conte*t

44、) throws IOE*ception, InterruptedE*ception sum2+; String str = value.toString().split(t); int rank = Integer.parseInt(str3); if(rank11) sum1=sum1+1; public static void main(String args) throws IOE*ception, ClassNotFoundE*ception, InterruptedE*ception Configuration conf = new Configuration(); conf.se

45、t(fs.defaultFS, hdfs:/0:9000); Job job = Job.getInstance(conf, eight); job.setMapperClass(MyMapper.class); job.setOutputKeyClass(Te*t.class); job.setOutputValueClass(Te*t.class); job.setJarByClass(CountRank.class); FileInputFormat.addInputPath(job, new Path(/sogou/data/sogou.500w.utf8); Fi

46、leOutputFormat.setOutputPath(job, new Path(/sogou/data/CountRank); job.waitForpletion(true); System.out.println(sum1=+sum1+tsum2=+sum2); float percentage = (float)sum1/(float)sum2; System.out.println(Rank在10以的點擊次數(shù)占比： +percentage*100+%); 直接輸入URL查詢的比例public class CountURL public static int sum1 = 0; p

47、ublic static int sum2 = 0; public static class MyMapper e*tends Mapper Override protected void map(Object key, Te*t value, Mapper.Conte*t conte*t) throws IOE*ception, InterruptedE*ception String str = value.toString().split(t); Pattern p = Pattern.pile(); Matcher matcher = p.matcher(str2); matcher.f

48、ind(); try if(matcher.group()!=null) sum1+; sum2+; catch (E*ception e) sum2+; public static void main(String args) throws IOE*ception, ClassNotFoundE*ception, InterruptedE*ception Configuration conf = new Configuration(); conf.set(fs.defaultFS, hdfs:/0:9000); Job job = Job.getInstance(conf

49、, nine); job.setMapperClass(MyMapper.class); job.setOutputKeyClass(Te*t.class); job.setOutputValueClass(Te*t.class); job.setJarByClass(CountURL.class); FileInputFormat.addInputPath(job, new Path(/sogou/data/sogou.500w.utf8); FileOutputFormat.setOutputPath(job, new Path(/sogou/data/CountURL); job.wai

50、tForpletion(true); System.out.println(sum1=+sum1+tsum2=+sum2); float percentage = (float)sum1/(float)sum2; System.out.println(直接用url%查詢的用戶占比： +percentage*100+%); 查詢搜索過仙劍奇?zhèn)b傳的uid，并且次數(shù)大于3public class CountUidGreater3 public static String Str=; public static int i=0; public static class Map e*tends Mapp

51、er Override protected void map(Object key, Te*t value, Mapper.Conte*t conte*t) throws IOE*ception, InterruptedE*ception String values=value.toString().split(t); String pattern=仙劍奇?zhèn)b傳; if(values2.equals(pattern) conte*t.write(new Te*t(values1), new IntWritable(1); public static class Reduce e*tends Re

52、ducer Override protected void reduce(Te*t key, Iterable value, Reducer.Conte*t conte*t) throws IOE*ception, InterruptedE*ception int sum=0; for(IntWritable v:value) sum=sum+v.get(); if(sum3) Str=Str+key.toString()+n; i+; public static void main(String args) Configuration conf=new Configuration(); co

53、nf.set(fs.defaultFS, hdfs:/0:9000); Job job = null; try job = Job.getInstance(conf, count); catch (IOE*ception e) / TODO Auto-generated catch block e.printStackTrace(); job.setJarByClass(CountUidGreater3.class); job.setMapperClass(Map.class); job.setReducerClass(Reduce.class); job.setOutpu

54、tKeyClass(Te*t.class); job.setOutputValueClass(IntWritable.class); try FileInputFormat.addInputPath(job, new Path(/sogou/data/sogou.500w.utf8); catch (IllegalArgumentE*ception e) / TODO Auto-generated catch block e.printStackTrace(); catch (IOE*ception e) / TODO Auto-generated catch block e.printSta

55、ckTrace(); try FileOutputFormat.setOutputPath(job, new Path(/sogou/data/CountUidGreater3); job.waitForpletion(true); catch (ClassNotFoundE*ception e) / TODO Auto-generated catch block e.printStackTrace(); catch (IOE*ception e) / TODO Auto-generated catch block e.printStackTrace(); catch (Interrupted

56、E*ception e) / TODO Auto-generated catch block e.printStackTrace(); System.out.println(i: +i); System.out.println(Str); 將4每步驟生成的結果保存到HDFS中使用INSERT OVERWRITE DIRECTORY可完成操作例如：將5生成的文件通過Java API方式導入到HBase一表將中5生成的文件通過Java API方式導入到HBase一表public class HBaseImport / reduce輸出的表名 private static String tableN

57、ame = test; / 初始化連接 static Configuration conf = null; static conf = HBaseConfiguration.create(); conf.set(hbase.rootdir, hdfs:/0:9000/hbase); conf.set(hbase.master, hdfs:/0:60000); conf.set(perty.clientPort, 2181); conf.set(hbase.zookeeper.quorum, master,slave1,slave2); conf.set(TableOutputFormat.OUTPUT_TABLE, tableName); public static class BatchMapper e*tends Mapper protected void map(LongWritable key, Te*t value, Mapper.Conte*t conte*t) th