演示文稿大數(shù)據(jù)采集與清洗

演示文稿大數(shù)據(jù)采集與清洗當(dāng)前第1頁(yè)\共有35頁(yè)\編于星期五\4點(diǎn)（優(yōu)選）大數(shù)據(jù)采集與清洗當(dāng)前第2頁(yè)\共有35頁(yè)\編于星期五\4點(diǎn)什么是大數(shù)據(jù)大數(shù)據(jù)處理流程大數(shù)據(jù)的主要特征大數(shù)據(jù)采集的概念大數(shù)據(jù)采集應(yīng)用當(dāng)前第3頁(yè)\共有35頁(yè)\編于星期五\4點(diǎn)1什么是大數(shù)據(jù)當(dāng)前第4頁(yè)\共有35頁(yè)\編于星期五\4點(diǎn)當(dāng)前第5頁(yè)\共有35頁(yè)\編于星期五\4點(diǎn)當(dāng)前第6頁(yè)\共有35頁(yè)\編于星期五\4點(diǎn)當(dāng)前第7頁(yè)\共有35頁(yè)\編于星期五\4點(diǎn)淘寶推薦

依據(jù)購(gòu)物行為偏好引薦依據(jù)你最近的閱讀行為和消費(fèi)行為進(jìn)行引薦依據(jù)你用的設(shè)備往來(lái)不斷猜特征.依據(jù)時(shí)節(jié)改變進(jìn)行引薦當(dāng)前第8頁(yè)\共有35頁(yè)\編于星期五\4點(diǎn)2014-032015-082017-102016-032018大數(shù)據(jù)工作首先寫(xiě)入政府工作報(bào)告『十三五規(guī)劃綱要』提出『實(shí)施國(guó)家大數(shù)據(jù)戰(zhàn)略』』2018年《政府工作報(bào)告》提出：實(shí)施大數(shù)據(jù)發(fā)展行動(dòng)，注重用互聯(lián)網(wǎng)、大數(shù)據(jù)等提升監(jiān)管效能國(guó)務(wù)院發(fā)布《促進(jìn)大數(shù)據(jù)發(fā)展的行動(dòng)綱要》十九大提出推動(dòng)大數(shù)據(jù)戰(zhàn)略，與實(shí)體經(jīng)濟(jì)深度融合行業(yè)現(xiàn)狀與前景當(dāng)前第9頁(yè)\共有35頁(yè)\編于星期五\4點(diǎn)當(dāng)前第10頁(yè)\共有35頁(yè)\編于星期五\4點(diǎn)2019年人社部擬最新發(fā)布15項(xiàng)新職業(yè)1.大數(shù)據(jù)工程技術(shù)人員2.云計(jì)算工程技術(shù)人員3.人工智能工程技術(shù)人員4.物聯(lián)網(wǎng)工程技術(shù)人員5....當(dāng)前第11頁(yè)\共有35頁(yè)\編于星期五\4點(diǎn)當(dāng)前第12頁(yè)\共有35頁(yè)\編于星期五\4點(diǎn)當(dāng)前第13頁(yè)\共有35頁(yè)\編于星期五\4點(diǎn)什么是大數(shù)據(jù)大數(shù)據(jù)（BigData）是指無(wú)法使用傳統(tǒng)和常用的軟件技術(shù)和工具在一定時(shí)間內(nèi)完成獲取、管理和處理的數(shù)據(jù)集當(dāng)前第14頁(yè)\共有35頁(yè)\編于星期五\4點(diǎn)2大數(shù)據(jù)的主要特征當(dāng)前第15頁(yè)\共有35頁(yè)\編于星期五\4點(diǎn)大數(shù)據(jù)主要特征

VolumeVelocityVarietyVeracity真實(shí)性（Veracity），即追求高質(zhì)量的數(shù)據(jù)。容量大（Volume）,指大規(guī)模的數(shù)據(jù)量，并且數(shù)據(jù)量呈持續(xù)增長(zhǎng)趨勢(shì)。速度快（Velocity）,指的是數(shù)據(jù)被創(chuàng)建和移動(dòng)的速度。種類多（Variety）,指數(shù)據(jù)來(lái)自多種數(shù)據(jù)源，數(shù)據(jù)種類和格式。Value價(jià)值密度低（Value），指隨著數(shù)據(jù)量的增長(zhǎng)，數(shù)據(jù)中有意義的信息卻沒(méi)有成相應(yīng)比例增長(zhǎng)。當(dāng)前第16頁(yè)\共有35頁(yè)\編于星期五\4點(diǎn)3大數(shù)據(jù)處理流程當(dāng)前第17頁(yè)\共有35頁(yè)\編于星期五\4點(diǎn)大數(shù)據(jù)處理流程

數(shù)據(jù)預(yù)處理就是將采集來(lái)的數(shù)據(jù)從多種數(shù)據(jù)庫(kù)導(dǎo)入到大型的分布式數(shù)據(jù)庫(kù)中（目前主要是hfds或hive）,并同時(shí)做一些簡(jiǎn)單的清洗和預(yù)處理工作。數(shù)據(jù)統(tǒng)計(jì)分析就是對(duì)上面已經(jīng)完成的存儲(chǔ)在大型分布式數(shù)據(jù)庫(kù)中的數(shù)據(jù)進(jìn)行歸類統(tǒng)計(jì)，可以滿足一般場(chǎng)景的分析需求。數(shù)據(jù)挖掘是對(duì)數(shù)據(jù)進(jìn)行基于各種算法的分析計(jì)算，從而起到預(yù)測(cè)的效果，實(shí)現(xiàn)一些高級(jí)別數(shù)據(jù)分析的需求。數(shù)據(jù)采集就是利用多種數(shù)據(jù)庫(kù)（關(guān)系型，NOSQL）去存儲(chǔ)不同來(lái)源的數(shù)據(jù)。數(shù)據(jù)展示就是對(duì)以上處理完的結(jié)果進(jìn)行分析，或者形成報(bào)表。當(dāng)前第18頁(yè)\共有35頁(yè)\編于星期五\4點(diǎn)4大數(shù)據(jù)采集的概念當(dāng)前第19頁(yè)\共有35頁(yè)\編于星期五\4點(diǎn)大數(shù)據(jù)采集的概念3、大數(shù)據(jù)采集技術(shù)方法大數(shù)據(jù)采集技術(shù)就是對(duì)數(shù)據(jù)進(jìn)行ETL操作，通過(guò)對(duì)數(shù)據(jù)進(jìn)行提取、轉(zhuǎn)換、加載，最終挖掘數(shù)據(jù)的潛在價(jià)值。ETL指的是Extract-Transform-Load，也就是抽取、轉(zhuǎn)換、加載。抽取->從各種數(shù)據(jù)源獲取數(shù)據(jù)轉(zhuǎn)換->按需求格式將源數(shù)據(jù)轉(zhuǎn)換為目標(biāo)數(shù)據(jù)加載->把目標(biāo)數(shù)據(jù)加載到數(shù)據(jù)倉(cāng)庫(kù)中2、數(shù)據(jù)采集與大數(shù)據(jù)采集的區(qū)別傳統(tǒng)數(shù)據(jù)采集：來(lái)源單一，數(shù)據(jù)量相當(dāng)??；結(jié)構(gòu)單一；關(guān)系數(shù)據(jù)庫(kù)和并行數(shù)據(jù)庫(kù)大數(shù)據(jù)的數(shù)據(jù)采集：來(lái)源廣泛，數(shù)量巨大；數(shù)據(jù)類型豐富；分布式數(shù)據(jù)庫(kù)1、什么是數(shù)據(jù)采集數(shù)據(jù)采集就是數(shù)據(jù)獲取，數(shù)據(jù)源主要分為線上數(shù)據(jù)和內(nèi)容數(shù)據(jù)當(dāng)前第20頁(yè)\共有35頁(yè)\編于星期五\4點(diǎn)大數(shù)據(jù)采集系統(tǒng)1.日志采集系統(tǒng)（ApacheFlume、Scribe）3.數(shù)據(jù)庫(kù)采集系統(tǒng)（關(guān)系型、nosql等各種數(shù)據(jù)庫(kù)）2.網(wǎng)絡(luò)數(shù)據(jù)采集系統(tǒng)（Scrapy框架、ApacheNutch）當(dāng)前第21頁(yè)\共有35頁(yè)\編于星期五\4點(diǎn)5大數(shù)據(jù)采集應(yīng)用當(dāng)前第22頁(yè)\共有35頁(yè)\編于星期五\4點(diǎn)當(dāng)前第23頁(yè)\共有35頁(yè)\編于星期五\4點(diǎn)技能準(zhǔn)備Python基礎(chǔ)Linux操作系統(tǒng)基本操作數(shù)據(jù)庫(kù)基礎(chǔ)（SQL語(yǔ)句操作）當(dāng)前第24頁(yè)\共有35頁(yè)\編于星期五\4點(diǎn)環(huán)境準(zhǔn)備PythonJdk(java環(huán)境)數(shù)據(jù)庫(kù)（mysql）當(dāng)前第25頁(yè)\共有35頁(yè)\編于星期五\4點(diǎn)Thanks

當(dāng)前第26頁(yè)\共有35頁(yè)\編于星期五\4點(diǎn)YOURTITLE

Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.當(dāng)前第27頁(yè)\共有35頁(yè)\編于星期五\4點(diǎn)YOURTITLE

Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.當(dāng)前第28頁(yè)\共有35頁(yè)\編于星期五\4點(diǎn)2OKPPT工作室當(dāng)前第29頁(yè)\共有35頁(yè)\編于星期五\4點(diǎn)YOURTITLE

Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.當(dāng)前第30頁(yè)\共有35頁(yè)\編于星期五\4點(diǎn)YOURTITLE

Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.當(dāng)前第31頁(yè)\共有35頁(yè)\編于星期五\4點(diǎn)YOURTITLE

21%9%28%42%當(dāng)前第32頁(yè)\共有35頁(yè)\編于星期五\4點(diǎn)3OKPPT工作室當(dāng)前第33頁(yè)\共有35頁(yè)\編于星期五\4點(diǎn)YOURTITLE

Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.當(dāng)前第34頁(yè)\共有35頁(yè)\編于星期五\4點(diǎn)YOURTITLE

Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdifficulttothemanwhowilltry.Nothingisdif