Python-爬虫-处理数据-BeautifulSoup4

爱被打了一巴掌 2024-04-18 08:51 63阅读 0赞

## bs4与lxml的区别： ##

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9odWFuZ2hhaXRhby5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70][]

## 解析工具对比： ##

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9odWFuZ2hhaXRhby5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 1][]

bs4：解析速度最慢，使用难度最简单

lxml：解析速度中，使用难度中

正则：解析速度快，使用难度难

## BS4使用： ##

获取补全的DOM树

from bs4 import BeautifulSoup
 
 html = """
 <html><head><title>The Dormouse's story</title></head>
 <body>
 The Dormouse's story
 Once upon a time there were three little sisters; and their names were
 <a href="http://example.com/elsie" class="sister" id="link1"></a>,
 <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
 <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
 and they lived at the bottom of a well.
 ...
 """
 
 soup = BeautifulSoup(html,'lxml')
 print(soup.prettify())

对DOM树的常用解析：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9odWFuZ2hhaXRhby5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 2][]

代码：

from bs4 import BeautifulSoup
 
 html = """
 <html><head><title>The Dormouse's story</title></head>
 <body>
 
 Once upon a time there were three little sisters; and their names were
 <a href="http://example.com/elsie" class="sister" id="link1"></a>,
 <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
 <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
 and they lived at the bottom of a well.
 ...
 
 """
 
 soup = BeautifulSoup(html,'lxml')
 print(soup.title)
 print(soup.p)

运行结果：

![20190904115200659.png][]

只找到第一个标签就返回

代码：

运行结果：

![20190904115509124.png][]

能够发现都是返回一个Tag对象

总结：都可以用bs4.【标签名】获取到第一个符合标签的DOM元素值

代码：

运行结果：

![2019090515143594.png][]

获取p标签的class，soup.p\['class'\]和soup.p.get('class')获取方式结果一样

## 遍历文档树： ##

contents和children

代码：

from bs4 import BeautifulSoup
 
 html = """
 <html><head><title>The Dormouse's story</title></head>
 <body>
 
 Once upon a time there were three little sisters; and their names were
 <a href="http://example.com/elsie" class="sister" id="link1"></a>,
 <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
 <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
 and they lived at the bottom of a well.
 ...
 
 
 
 """
 
 soup = BeautifulSoup(html,'lxml')
 
 head_tag = soup.head
 print(head_tag.contents)
 print(head_tag.children)
 for i in head_tag.children:
 print(i)

运行结果：

![2019090515324493.png][]

contents是返回数组形式，children是返回一个迭代器，但是结果一致

string和strings和stripped\_strings

代码：

from bs4 import BeautifulSoup
 
 html = """
 <html><head><title>The Dormouse's story</title></head>
 <body>
 
 Once upon a time there were three little sisters; and their names were
 <a href="http://example.com/elsie" class="sister" id="link1"></a>,
 <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
 <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
 and they lived at the bottom of a well.
 ...
 
 
 
 """
 
 soup = BeautifulSoup(html,'lxml')
 
 for string in soup.strings:
 print(repr(string))
 
 
 for string in soup.stripped_strings:
 print(repr(string))
 
 
 print(soup.b.strings)

string：获取某个标签下的非标签字符串，返回来的是一个字符串，如果这个标签下有多行字符，那么久不能获取到了。

strings：后去某个标签下的子孙非标签字符串，返回来的是一个生成器。

stripped\_strings：获取某个标签下的子孙非标签字符串，会去掉空白字符，返回来的是一个生成器。

## 搜索文档树： ##

find和find\_all方法：

find方法是找到第一个满足条件的标签后就立刻返回，只返回一个元素。

find\_all方法是吧所有满足条件的标签都返回，返回的是一个列表

select方法（推荐）类似find\_all：

#--coding:utf-8--
 from bs4 import BeautifulSoup
 
 html = """
 <html><head><title>The Dormouse's story</title></head>
 
 The Dormouse's story
 
 Once upon a time there were three little sisters; and their names were
 <a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
 <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
 <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
 and they lived at the bottom of a well.
 
 ...
 """
 
 soup = BeautifulSoup(html,'lxml')
 
 # （1）通过标签名查找：
 print(soup.select('a'))
 # （2）通过类名查找：
 print(soup.select('.sister'))
 # （3）通过id查找：
 print(soup.select('#link1'))
 # （4）组合查找：
 print(soup.select('p #link1'))
 print(soup.select('head > title'))
 # （5）通过属性查找：
 print(soup.select('a[href="http://example.com/elsie"]'))
 # （6）获取内容：
 print(soup.select('title')[0].get_text())

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9odWFuZ2hhaXRhby5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/8d8f768d5b82403eb655dc0220e06163.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9odWFuZ2hhaXRhby5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 1]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/f5c182743e9e45d89dfa93cd4d1b86ae.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9odWFuZ2hhaXRhby5ibG9nLmNzZG4ubmV0_size_16_color_FFFFFF_t_70 2]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/2c31ad9a846d431396dacb91b3a68ddb.png
[20190904115200659.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/af04bc406d8d4a23ad0f827ebce7c3b0.png
[20190904115509124.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/97c9676b1e514d8f96088b94a822bd03.png
[2019090515143594.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/30ab71e75aa94f579cbab42fc3e1ccf8.png
[2019090515324493.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/17/3b35f4b610774d6dade7dd15bd2f7a5c.png