Python爬蟲程序設(shè)計KC23

1、2.3.1 BeautifulSoup查找HTML元素2.3.1 BeautifulSoup查找HTML元素查找文檔的元素是我們爬取網(wǎng)頁信息的重要手段，BeautifulSoup提供了一系列的查找元素的方法，其中功能強大的find_all函數(shù)就是其中常用的一個方法。find_all函數(shù)的原型如下：find_all(self, name=None, attrs=, recursive=True, text=None, limit=None, *kwargs)self表明它是一個類成員函數(shù)；name是要查找的tag元素名稱，默認是None，如果不提供，就是查找所有的元素；attrs是元素的屬性，它

2、是一個字典，默認是空，如果提供就是查找有這個指定屬性的元素；recursive指定查找是否在元素節(jié)點的子樹下面全范圍進行，默認是True；后面的text、limit、kwargs參數(shù)比較復(fù)雜，將在后面用到時介紹；find_all函數(shù)返回查找到的所有指定的元素的列表，每個元素是一個bs4.element.Tag對象。find_all函數(shù)是查找所有滿足要求的元素節(jié)點，如果我們只查找一個元素節(jié)點，那么可以使用find函數(shù)，它的原型如下：find(self, name=None, attrs=, recursive=True, text=None, limit=None, *kwargs)使用方法與f

3、ind_all類似，不同的是它只返回第一個滿足要求的節(jié)點，不是一個列表。例2-3-1：查找文檔中的元素from bs4 import BeautifulSoupdoc=The Dormouses storyThe Dormouses storyOnce upon a time there were three little sisters; and their names wereElsie,Lacie andTillie;and they lived at the bottom of a well.soup=BeautifulSoup(doc,lxml)tag=soup.find(title

4、)print(type(tag),tag)程序結(jié)果： The Dormouses story由此可見查找到元素，元素類型是一個bs4.element.Tag對象。例2-3-2：查找文檔中的所有元素from bs4 import BeautifulSoupdoc=The Dormouses storyThe Dormouses storyOnce upon a time there were three little sisters; and their names wereElsie,Lacie andTillie;and they lived at the bottom of a well.

5、soup=BeautifulSoup(doc,lxml)tags=soup.find_all(a)for tag in tags: print(tag)程序結(jié)果找到3個元素：ElsieLacieTillie例2-3-3：查找文檔中的第一個元素from bs4 import BeautifulSoupdoc=The Dormouses storyThe Dormouses storyOnce upon a time there were three little sisters; and their names wereElsie,Lacie andTillie;and they lived a

6、t the bottom of a well.soup=BeautifulSoup(doc,lxml)tag=soup.find(a)print(tag)程序結(jié)果找到第一個元素：Elsie例2-3-4：查找文檔中class=title的元素from bs4 import BeautifulSoupdoc=The Dormouses storyThe Dormouses storyOnce upon a time there were three little sisters; and their names wereElsie,Lacie andTillie;and they lived at

7、 the bottom of a well.soup=BeautifulSoup(doc,lxml)tag=soup.find(p,attrs=class:title)print(tag)程序結(jié)果找到class=title的元素The Dormouses story很顯然如果使用：tag=soup.find(p)也能找到這個元素，因為它是文檔的第一個元素。例2-3-5：查找文檔中class=sister的元素from bs4 import BeautifulSoupdoc=The Dormouses storyThe Dormouses storyOnce upon a time there

8、were three little sisters; and their names wereElsie,Lacie andTillie;and they lived at the bottom of a well.soup=BeautifulSoup(doc,lxml)tags=soup.find_all(name=None,attrs=class:sister)for tag in tags: print(tag)其中name=None表示無論是什么名字的元素，程序結(jié)果找到3個：a class=sister href=/elsie id=link1ElsieLacieTillie對于這個文

9、檔，很顯然語句：tags=soup.find_all(a)或者：tags=soup.find_all(a,attrs=class:sister)效果一樣。2.3.2 BeautifulSoup獲取元素的屬性值2.3.2 BeautifulSoup獲取元素的屬性值如果一個元素已經(jīng)找到，例如找到元素，那么怎么樣獲取它的屬性值呢？BeautifulSoup使用:tagattrName來獲取tag元素的名稱為attrName的屬性值，其中tag是一個bs4.element.Tag對象。例2-3-6：查找文檔中所有超級鏈接地址from bs4 import BeautifulSoupdoc=The Do

10、rmouses storyThe Dormouses storyOnce upon a time there were three little sisters; and their names wereElsie,Lacie andTillie;and they lived at the bottom of a well.soup=BeautifulSoup(doc,lxml)tags=soup.find_all(a)for tag in tags: print(taghref)程序結(jié)果：/elsie/lacie/tillie2.3.3 BeautifulSoup獲取元素包含的文本值2.3.

11、3 BeautifulSoup獲取元素包含的文本值如果一個元素已經(jīng)找到，例如找到元素，那么怎么樣獲取它包含的文本值呢？BeautifulSoup使用:tag.text來獲取tag元素包含的文本值，其中tag是一個bs4.element.Tag對象。例2-3-7：查找文檔中所有超級鏈接包含的文本值from bs4 import BeautifulSoupdoc=The Dormouses storyThe Dormouses storyOnce upon a time there were three little sisters; and their names wereElsie,Lacie

12、 andTillie;and they lived at the bottom of a well.soup=BeautifulSoup(doc,lxml)tags=soup.find_all(a)for tag in tags: print(tag.text)程序結(jié)果：ElsieLacieTillie例2-3-8：查找文檔中所有超級鏈接包含的文本值from bs4 import BeautifulSoupdoc=The Dormouses storyThe Dormouses storyOnce upon a time there were three little sisters; and

13、 their names wereElsie,Lacie andTillie;and they lived at the bottom of a well.soup=BeautifulSoup(doc,lxml)tags=soup.find(p)for tag in tags: print(tag.text)程序結(jié)果：The Dormouses storyOnce upon a time there were three little sisters; and their names wereElsie,Lacie andTillie;and they lived at the bottom

14、of a well.其中第二個包含的值就是節(jié)點子樹下面所有文本節(jié)點的組合值。2.3.4 BeautifulSoup高級查找一般find或者find_all都能滿足我們的需要，如果還不能那么可以設(shè)計一個查找函數(shù)來進行查找。例2-3-9：我們查找文檔中的href=/lacie的節(jié)點元素from bs4 import BeautifulSoupdoc=The Dormouses storyElsieLacieTilliedef myFilter(tag): print() return (=a and tag.has_attr(href) and taghref=/lacie)soup=Beauti

15、fulSoup(doc,lxml)tag=soup.find_all(myFilter)print(tag)程序結(jié)果：htmlheadtitlebodyaaaLacie 說明：在程序中我們定義了一個篩選函數(shù)myFilter(tag)，它的參數(shù)是tag對象，在調(diào)用soup.find_all(myFilter)時程序會把每個tag元素傳遞給myFilter函數(shù)，由該函數(shù)決定這個tag的取舍，如果myFilter返回True就保留這個tag到結(jié)果集中，不然就丟掉這個tag。因此程序執(zhí)行時可以看到html,body,head,title,body,a,a,a等一個個tag經(jīng)過myFilter的篩選，只

16、有節(jié)點Lacie滿足要求，因此結(jié)果為：Lacie其中：是tag的名稱；tag.has_attr(attName)判斷tag是否有attName屬性；tagattName是tag的attName屬性值；例2-3-10：通過函數(shù)查找可以查找到一些復(fù)雜的節(jié)點元素，查找文本值以cie結(jié)尾所有節(jié)點from bs4 import BeautifulSoupdoc=The Dormouses storyElsieLacieTillieTilciedef endsWith(s,t): if len(s)=len(t): return slen(s)-len(t):=t return Falsedef myFilter(ta