【移動(dòng)應(yīng)用開發(fā)技術(shù)】實(shí)例：使用puppeteer headless方式抓取JS網(wǎng)

【移動(dòng)應(yīng)用開發(fā)技術(shù)】實(shí)例：使用puppeteerheadless方式抓取JS網(wǎng)

puppeteergooglechrome團(tuán)隊(duì)出品的puppeteer是依賴nodejs和chromium的自動(dòng)化測試庫，它的最大優(yōu)點(diǎn)就是可以處理網(wǎng)頁中的動(dòng)態(tài)內(nèi)容，如JavaScript，能夠更好的模擬用戶。有些網(wǎng)站的反爬蟲手段是將部分內(nèi)容隱藏于某些javascript/ajax請求中，致使直接獲取a標(biāo)簽的方式不奏效。甚至有些網(wǎng)站會(huì)設(shè)置隱藏元素“陷阱”，對用戶不可見，腳本觸發(fā)則認(rèn)為是機(jī)器。這種情況下，puppeteer的優(yōu)勢就凸顯出來了。它可實(shí)現(xiàn)如下功能：開源地址：[/GoogleChrome/puppeteer/][1]npmipuppeteer注意先安裝nodejs,并在nodejs文件根目錄下執(zhí)行（npm文件同級）。安裝過程中會(huì)下載chromium，大約120M。用兩天（大約10小時(shí)）摸索，繞過了相當(dāng)多的異步的坑，筆者對puppeteer和nodejs有了一定的掌握。一張長圖，抓取blog文章列表：抓取blog文章以csdnblog為例，文章內(nèi)容需要點(diǎn)擊“閱讀全文”來獲取，這就導(dǎo)致只能讀取dom的腳本失效。/**

*loadarticletolocalfiles

**/

constpuppeteer=require('puppeteer');

//emulateiphone

constuserAgent='Mozilla/5.0(iPhone;CPUiPhoneOS11_0likeMacOSX)AppleWebKit/604.1.38(KHTML,likeGecko)Version/11.0Mobile/15A372Safari/604.1';

constworkPath='./contents';

constfs=require("fs");

if(!fs.existsSync(workPath)){

fs.mkdirSync(workPath)

}

//baseurl

constrootUrl='/';

//maxwaitmilliseconds

constmaxWait=100;

//maxloopscrolltimes

constmakLoop=10;

(async()=>{

leturl;

letcountUrl=0;

constbrowser=awaitpuppeteer.launch({headless:false});//setheadless:truewillhidechromiumUI

constpage=awaitbrowser.newPage();

awaitpage.setUserAgent(userAgent);

awaitpage.setViewport({width:414,height:736});

awaitpage.setRequestInterception(true);

//filtertoblockimages

page.on('request',request=>{

if(request.resourceType()==='image')

request.abort();

else

request.continue();

});

awaitpage.goto(rootUrl);

for(leti=0;i<makLoop;i++){

try{

awaitpage.evaluate(()=>window.scrollTo(0,document.body.scrollHeight));

awaitpage.waitForNavigation({timeout:maxWait,waitUntil:['networkidle0']});

}catch(err){

console.log('scrolltobottomandthenwait'+maxWait+'ms.');

}

}

awaitpage.screenshot({path:workPath+'/screenshot.png',fullPage:true,quality:100,type:'jpeg'});

//#feedlist_idli[data-type="blog"]a

constsel='#feedlist_idli[data-type="blog"]h3a';

consthrefs=awaitpage.evaluate((sel)=>{

letelements=Array.from(document.querySelectorAll(sel));

letlinks=elements.map(element=>{

returnelement.href

})

returnlinks;

},sel);

console.log('totallinks:'+hrefs.length);

process();

asyncfunctionprocess(){

if(countUrl<hrefs.length){

url=hrefs[countUrl];

countUrl++;

}else{

browser.close();

return;

}

console.log('processingurl:'+url);

try{

consttab=awaitbrowser.newPage();

awaittab.setUserAgent(userAgent);

awaittab.setViewport({width:414,height:736});

awaittab.setRequestInterception(true);

//filtertoblockimages

tab.on('request',request=>{

if(request.resourceType()==='image')

request.abort();

else

request.continue();

});

awaittab.goto(url);

//executetaprequest

try{

awaittab.tap('.read_more_btn');

}catch(err){

console.log('there\'snonereadmorebutton.NoneedtoTAP');

}

lettitle=awaittab.evaluate(()=>document.querySelector('#article.article_title').innerText);

letcontents=awaittab.evaluate(()=>document.querySelector('#article.article_content').innerText);

contents='TITLE:'+title+'\nURL:'+url+'\nCONTENTS:\n'+contents;

constfs=require("fs");

fs.writeFileSync(workPath+'/'+tab.url().substring(tab.url().lastIndexOf('/'),tab.url().length)+'.txt',contents);

console.log(title+"hasbeendownloadedtolocal.");

awaittab.close();

}catch(err){

console.log('url:'+tab.url()+'\n'+err.toString());

}finally{

process();

}

}

})();

執(zhí)行過程錄屏可以在我公眾號查看，下邊是截圖：執(zhí)行結(jié)果文章內(nèi)容列表：文章內(nèi)容：結(jié)束語以前就想過既然nodejs是使用JavaScript腳本語言，那么它肯定能處理網(wǎng)頁的JavaScript內(nèi)容，但并沒有發(fā)現(xiàn)合適的/高效率的庫。直到發(fā)現(xiàn)puppeteer，才下定決心試水。話說回來，nodejs的異步真的是很頭疼的一件事，這上百行代碼我竟然折騰了10個(gè)小時(shí)。大家可拓展下代碼中process()方法，使用async.eachSeries，我使用的遞歸方式并不是最優(yōu)解。事實(shí)上，逐一處理并不高效，原本我寫了一個(gè)異步的關(guān)閉browser方法：lettryCloseBrowser=setInterval(function(){

console.log("checkifanyprocessrunning...")

if(countDown<=0){

clearInterval(tryCloseBrowser);

console.log("noneprocessrunning,close.")

browser.close();

}

},3000);按照這個(gè)思路