BeautifulSoup爬取数据常用方法总结

╰+攻爆jí腚メ 2022-12-16 03:54 127阅读 0赞

Beautiful Soup 是一个可以从HTML或XML文件中提取数据的Python库.它能够通过你喜欢的转换器实现惯用的文档导航,查找,修改文档的方式.Beautiful Soup会帮你节省数小时甚至数天的工作时间.

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQ1MTc2NTQ4_size_16_color_FFFFFF_t_70]

### 文章目录 ###

* 安装BeautifulSoup
 * * 几个简单的浏览结构化数据的方法
 * 从文档中找到所有的< a>标签的链接
 * 在文档中获取所有的文字内容
 * 常见解释器的优缺点
 * Tag
 * * Name
 * Attributes
 * 可以遍历的字符串
 * BeautifulSoup
 * 注释及特殊字符串
 * 遍历文档树
 * * 子节点
 * * \- find\_all
 * .contents和.children

# 安装BeautifulSoup #

pip3 install -i https://pypi.tuna.tsinghua.edu.cn/simple beautifulsoup4

from bs4 import BeautifulSoup

*  素材

html_doc = """
 <html><head><title>The Dormouse's story</title></head>
 <body>
 The Dormouse's story
 
 Once upon a time there were three little sisters; and their names were
 <a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
 <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
 <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
 and they lived at the bottom of a well.
 
 ...
 """

soup = BeautifulSoup(html_doc,"lxml")

## 几个简单的浏览结构化数据的方法 ##

soup.title

<title>The Dormouse's story</title>

soup.title.name

'title'

soup.title.string

"The Dormouse's story"

soup.title.text

"The Dormouse's story"

soup.title.parent.name

'head'

soup.p

The Dormouse's story

soup.p.name

'p'

soup.p["class"]

['title']

soup.a

<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>

soup.find("a")

<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>

soup.find_all("a")

[<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
 <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a>,
 <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>]

## 从文档中找到所有的< a>标签的链接 ##

for link in soup.find_all("a"):
        print(link.get("href"))

http://example.com/elsie
    http://example.com/lacie
    http://example.com/tillie

## 在文档中获取所有的文字内容 ##

print(soup.get_text())

The Dormouse's story
    
    The Dormouse's story
    Once upon a time there were three little sisters; and their names were
    Elsie,
    Lacie and
    Tillie;
    and they lived at the bottom of a well.
    ...

## 常见解释器的优缺点 ##

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQ1MTc2NTQ4_size_16_color_FFFFFF_t_70_pic_center]

# Tag #

*  Tag有很多方法和属性,在 遍历文档树 和 搜索文档树 中有详细解释.现在介绍一下tag中最重要的属性: name和attributes

soup = BeautifulSoup('Extremely bold')
 tag = soup.b
 tag

Extremely bold

type(tag)

bs4.element.Tag

## Name ##

*  每个tag都有自己的名字,通过 .name 来获取:

tag.name

'b'

*  如果改变了tag的name,那将影响所有通过当前Beautiful Soup对象生成的HTML文档

tag.name = "blockquote"
    tag

<blockquote class="boldest">Extremely bold</blockquote>

## Attributes ##

*  一个tag可能有很多个属性.tag  **有一个 “class” 的属性,值为 “boldest” . tag的属性的操作方法与字典相同:**

tag["class"]

['boldest']

tag.attrs

{'class': ['boldest']}

*  tag的属性可以被添加,删除或修改. 再说一次, tag的属性操作方法与字典一样

tag["class"] = "verybold"
    tag["id"] = 1
    tag

<blockquote class="verybold" id="1">Extremely bold</blockquote>

del tag["class"]
    tag

<blockquote id="1">Extremely bold</blockquote>

***多值属性***

css_soup = BeautifulSoup('')
 css_soup.p['class']

['body', 'strikeout']

css_soup = BeautifulSoup('')
 css_soup.p['class']

['body']

# 可以遍历的字符串 #

1.  字符串常被包含在tag内.Beautiful Soup用 NavigableString 类来包装tag中的字符串:

tag.string

'Extremely bold'

type(tag.string)

bs4.element.NavigableString

1.  一个 NavigableString 字符串与Python中的Unicode字符串相同,  
    并且还支持包含在遍历文档树 和 搜索文档树 中的一些特性.  
    通过 unicode() 方法可以直接将 NavigableString 对象转换成Unicode字符串:
2.  tag中包含的字符串不能编辑，但是可以被替换成其他的字符串，用replace\_with()方法

tag.string.replace_with("No longer bold")
    tag

<blockquote id="1">No longer bold</blockquote>

# BeautifulSoup #

*  BeautifulSoup 对象表示的是一个文档的全部内容.大部分时候,可以把它当作 Tag 对象,它支持 遍历文档树 和 搜索文档树 中描述的大部分的方法.

--------------------

*  因为 BeautifulSoup 对象并不是真正的HTML或XML的tag,所以它没有name和attribute属性.但有时查看它的 .name 属性是很方便的,所以 BeautifulSoup 对象包含了一个值为 “\[document\]” 的特殊属性 .name

soup.name

'[document]'

# 注释及特殊字符串 #

1.  文档的注释部分

markup = ""
 soup = BeautifulSoup(markup)
 comment = soup.b.string
 comment

'Hey, buddy. Want to buy a used parser?'

type(comment)

bs4.element.Comment

1.  Comment 对象是一个特殊类型的 NavigableString 对象:

comment

'Hey, buddy. Want to buy a used parser?'

但是当它出现在HTML文档中时, Comment 对象会使用特殊的格式输出:

print(soup.prettify())

from bs4 import CData
    cdata = CData("A CDATA block")
    comment.replace_with(cdata)
    print(soup.b.prettify())

<![CDATA[A CDATA block]]>

# 遍历文档树 #

html_doc = """
 <html><head><title>The Dormouse's story</title></head>
 <body>
 The Dormouse's story
 
 Once upon a time there were three little sisters; and their names were
 <a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>,
 <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and
 <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>;
 and they lived at the bottom of a well.
 
 ...
 """

from bs4 import BeautifulSoup

soup = BeautifulSoup(html_doc,"html.parser")

## 子节点 ##

一个Tag可能包含多个字符串或其它的Tag,这些都是这个Tag的子节点.Beautiful Soup提供了许多操作和遍历子节点的属性.

soup.head

<head><title>The Dormouse's story</title></head>

soup.title

<title>The Dormouse's story</title>

这是个获取tag的小窍门,可以在文档树的tag中多次调用这个方法.下面的代码可以获取标签中的第一个**标签:**

soup.body.b

The Dormouse's story

通过点取属性的方式只能获得当前名字的第一个tag:

soup.a

<a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>

### \- find\_all ###

如果想要得到所有的标签,或是通过名字得到比一个tag更多的内容的时候,就需要用到 Searching the tree 中描述的方法,比如: find\_all()

soup.find_all("a")

### .contents和.children ###

head_tag = soup.head
    head_tag

<head><title>The Dormouse's story</title></head>

head_tag.contents

[<title>The Dormouse's story</title>]

head_tag.contents[0]

<title>The Dormouse's story</title>

head_tag.contents[0].contents

["The Dormouse's story"]

BeautifulSoup 对象本身一定会包含子节点,也就是说标签也是 BeautifulSoup 对象的子节点:

soup.contents

['\n',
 <html><head><title>The Dormouse's story</title></head>
 <body>
 The Dormouse's story
 Once upon a time there were three little sisters; and their names were
 <a class="sister" href="http://example.com/elsie" id="link1">Elsie</a>,
 <a class="sister" href="http://example.com/lacie" id="link2">Lacie</a> and
 <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a>;
 and they lived at the bottom of a well.
 ...
 </body></html>]

len(soup.contents)

soup.contents[0].name

![在这里插入图片描述][resize_m_lfit_w_962_pic_center]

> 到这里就结束了，如果对你有帮助，欢迎点赞关注评论，你的点赞对我很重要

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQ1MTc2NTQ4_size_16_color_FFFFFF_t_70 1]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQ1MTc2NTQ4_size_16_color_FFFFFF_t_70]: /images/20221123/32f01332ad9d483da5ccbd2154186ca0.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQ1MTc2NTQ4_size_16_color_FFFFFF_t_70_pic_center]: /images/20221123/4f4df89ab5214192b58a4348a0e48163.png
[resize_m_lfit_w_962_pic_center]: /images/20221123/60a25e437ba84b20b1cfc77badbcac95.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQ1MTc2NTQ4_size_16_color_FFFFFF_t_70 1]: /images/20221123/5e5bf5639e644b51bcc8d45545e3fcc1.png