爬虫之pyquery库

刺骨的言语ヽ痛彻心扉 2022-10-03 00:48 150阅读 0赞

官方文档：[https://pyquery.readthedocs.io/en/latest/][https_pyquery.readthedocs.io_en_latest]

PyQuery是一个强大又灵活的网页解析库。如果你觉得正则写起来太麻烦、BeautifulSoup语法太难记，而你熟悉jQury的语法，那么PyQuery就是你的绝佳选择。

## 一、开始 ##

**字符串初始化：**

from pyquery import PyQuery as pq
    d = pq("<html>哈哈哈</html>")  # 现在d就相当于jQuery的$
    print(d("html"))

**URL初始化：**

from pyquery import PyQuery as pq
    d = pq(url="https://www.baidu.com")
    print(d("head"))

**文件初始化：**

from pyquery import PyQuery as pq
    d = pq(filename='demo.html')  # filename指定文件路径
    print(d("head"))

## **二、**基本CSS选择器 ##

![ContractedBlock.gif][] ![ExpandedBlockStart.gif][]

html = """
    <div id="container">
        <ul class="list">
             <li class="item-0">first item</li>
             <li class="item-1"><a href="link2.html">second item</a></li>
             <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>
             <li class="item-1 active"><a href="link4.html">fourth item</a></li>
             <li class="item-0"><a href="link5.html">fifth item</a></li>
         </ul>
     </div>
    """
    from pyquery import PyQuery as pq
    d = pq(html)
    print(d("#container .list li"))

## 三、查找元素 ##

### 子元素 ###

d("css选择器").find("li")

![ContractedBlock.gif][] ![ExpandedBlockStart.gif][]

html = """
    <div id="container">
        <ul class="list">
             <li class="item-0">first item</li>
             <li class="item-1"><a href="link2.html">second item</a></li>
             <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>
             <li class="item-1 active"><a href="link4.html">fourth item</a></li>
             <li class="item-0"><a href="link5.html">fifth item</a></li>
         </ul>
     </div>
    """
    from pyquery import PyQuery as pq
    d = pq(html)
    items = d(".list")
    print(type(items))  # <class 'pyquery.pyquery.PyQuery'>
    li = items.find("li")
    print(type(li))    # <class 'pyquery.pyquery.PyQuery'>
    print(li)
    """
     <li class="item-0">first item</li>
     <li class="item-1"><a href="link2.html">second item</a></li>
     <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>
     <li class="item-1 active"><a href="link4.html">fourth item</a></li>
     <li class="item-0"><a href="link5.html">fifth item</a></li>
    """

### **父元素** ###

d("css选择器").parent(<css选择器(可无)>)

![ContractedBlock.gif][] ![ExpandedBlockStart.gif][]

html = """
    <div class="wrap">
        <div id="container">
            <ul class="list">
                 <li class="item-0">first item</li>
                 <li class="item-1"><a href="link2.html">second item</a></li>
                 <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>
                 <li class="item-1 active"><a href="link4.html">fourth item</a></li>
                 <li class="item-0"><a href="link5.html">fifth item</a></li>
             </ul>
        </div>
    </div>
    """
    from pyquery import PyQuery as pq
    d = pq(html)
    items = d(".list")
    parents = items.parents()
    print(parents)
    """
    <div class="wrap">
        <div id="container">
            <ul class="list">
                 <li class="item-0">first item</li>
                 <li class="item-1"><a href="link2.html">second item</a></li>
                 <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>
                 <li class="item-1 active"><a href="link4.html">fourth item</a></li>
                 <li class="item-0"><a href="link5.html">fifth item</a></li>
            </ul>
        </div>
    </div>
    <div id="container">
        <ul class="list">
             <li class="item-0">first item</li>
             <li class="item-1"><a href="link2.html">second item</a></li>
             <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>
             <li class="item-1 active"><a href="link4.html">fourth item</a></li>
             <li class="item-0"><a href="link5.html">fifth item</a></li>
        </ul>
    </div>
    """

d(".list").parents()

![ContractedBlock.gif][] ![ExpandedBlockStart.gif][]

html = """
    <div class="wrap">
        <div id="container">
            <ul class="list">
                 <li class="item-0">first item</li>
                 <li class="item-1"><a href="link2.html">second item</a></li>
                 <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>
                 <li class="item-1 active"><a href="link4.html">fourth item</a></li>
                 <li class="item-0"><a href="link5.html">fifth item</a></li>
             </ul>
        </div>
    </div>
    """
    from pyquery import PyQuery as pq
    d = pq(html)
    items = d(".list")
    parents = items.parents(".wrap")
    print(parents)
    """
    <div class="wrap">
        <div id="container">
            <ul class="list">
                 <li class="item-0">first item</li>
                 <li class="item-1"><a href="link2.html">second item</a></li>
                 <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>
                 <li class="item-1 active"><a href="link4.html">fourth item</a></li>
                 <li class="item-0"><a href="link5.html">fifth item</a></li>
            </ul>
        </div>
    </div>
    """

d(".list").parents(".wrap")

### 兄弟元素 ###

d("css选择器").siblings(<css选择器(可无)>)

![ContractedBlock.gif][] ![ExpandedBlockStart.gif][]

html = """
    <div class="wrap">
        <div id="container">
            <ul class="list">
                 <li class="item-0">first item</li>
                 <li class="item-1"><a href="link2.html">second item</a></li>
                 <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>
                 <li class="item-1 active"><a href="link4.html">fourth item</a></li>
                 <li class="item-0"><a href="link5.html">fifth item</a></li>
             </ul>
        </div>
    </div>
    """
    from pyquery import PyQuery as pq
    d = pq(html)
    li = d(".list .item-0.active")
    print(li.siblings())
    """
    <li class="item-1"><a href="link2.html">second item</a></li>
    <li class="item-0">first item</li>
    <li class="item-1 active"><a href="link4.html">fourth item</a></li>
    <li class="item-0"><a href="link5.html">fifth item</a></li>
    """

![ContractedBlock.gif][] ![ExpandedBlockStart.gif][]

html = """
    <div class="wrap">
        <div id="container">
            <ul class="list">
                 <li class="item-0">first item</li>
                 <li class="item-1"><a href="link2.html">second item</a></li>
                 <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>
                 <li class="item-1 active"><a href="link4.html">fourth item</a></li>
                 <li class="item-0"><a href="link5.html">fifth item</a></li>
             </ul>
        </div>
    </div>
    """
    from pyquery import PyQuery as pq
    d = pq(html)
    li = d(".list .item-0.active")
    print(li.siblings(".active"))
    """
    <li class="item-1 active"><a href="link4.html">fourth item</a></li>
    """

## 四、遍历 ##

![ContractedBlock.gif][] ![ExpandedBlockStart.gif][]

html = """
    <div class="wrap">
        <div id="container">
            <ul class="list">
                 <li class="item-0">first item</li>
                 <li class="item-1"><a href="link2.html">second item</a></li>
                 <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>
                 <li class="item-1 active"><a href="link4.html">fourth item</a></li>
                 <li class="item-0"><a href="link5.html">fifth item</a></li>
             </ul>
        </div>
    </div>
    """
    from pyquery import PyQuery as pq
    d = pq(html)
    li = d("li").items()
    print(type(li))  # <class 'generator'>
    for i in li:
        print(i)
    """
    <li class="item-0">first item</li>     
    <li class="item-1"><a href="link2.html">second item</a></li>           
    <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>            
    <li class="item-1 active"><a href="link4.html">fourth item</a></li>            
    <li class="item-0"><a href="link5.html">fifth item</a></li>
    """

## 五、获取信息 ##

### 获取属性 ###

![ContractedBlock.gif][] ![ExpandedBlockStart.gif][]

html = """
    <div class="wrap">
        <div id="container">
            <ul class="list">
                 <li class="item-0">first item</li>
                 <li class="item-1"><a href="link2.html">second item</a></li>
                 <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>
                 <li class="item-1 active"><a href="link4.html">fourth item</a></li>
                 <li class="item-0"><a href="link5.html">fifth item</a></li>
             </ul>
        </div>
    </div>
    """
    from pyquery import PyQuery as pq
    d = pq(html)
    a = d(".item-0.active a")
    print(a.attr("href"))
    print(a.attr.href)

### 获取文本 ###

![ContractedBlock.gif][] ![ExpandedBlockStart.gif][]

html = """
    <div class="wrap">
        <div id="container">
            <ul class="list">
                 <li class="item-0">first item</li>
                 <li class="item-1"><a href="link2.html">second item</a></li>
                 <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>
                 <li class="item-1 active"><a href="link4.html">fourth item</a></li>
                 <li class="item-0"><a href="link5.html">fifth item</a></li>
             </ul>
        </div>
    </div>
    """
    from pyquery import PyQuery as pq
    d = pq(html)
    a = d(".item-0.active a")
    print(a.text())
    """
    third item
    """

### 获取html ###

![ContractedBlock.gif][] ![ExpandedBlockStart.gif][]

html = """
    <div class="wrap">
        <div id="container">
            <ul class="list">
                 <li class="item-0">first item</li>
                 <li class="item-1"><a href="link2.html">second item</a></li>
                 <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>
                 <li class="item-1 active"><a href="link4.html">fourth item</a></li>
                 <li class="item-0"><a href="link5.html">fifth item</a></li>
             </ul>
        </div>
    </div>
    """
    from pyquery import PyQuery as pq
    d = pq(html)
    li = d(".item-0.active")
    print(li)
    print(li.html())
    """
    <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>        
    <a href="link3.html"><span class="bold">third item</span></a>
    """

## 六、DOM操作 ##

### addClass()、removeClass() ###

![ContractedBlock.gif][] ![ExpandedBlockStart.gif][]

html = """
    <div class="wrap">
        <div id="container">
            <ul class="list">
                 <li class="item-0">first item</li>
                 <li class="item-1"><a href="link2.html">second item</a></li>
                 <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>
                 <li class="item-1 active"><a href="link4.html">fourth item</a></li>
                 <li class="item-0"><a href="link5.html">fifth item</a></li>
             </ul>
        </div>
    </div>
    """
    from pyquery import PyQuery as pq
    d = pq(html)
    li = d(".item-0.active")
    print(li)
    li.removeClass("active")
    print(li)
    li.addClass("active")
    print(li)
    """
    <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>            
    <li class="item-0"><a href="link3.html"><span class="bold">third item</span></a></li>            
    <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>
    """

### attr()、css() ###

![ContractedBlock.gif][] ![ExpandedBlockStart.gif][]

html = """
    <div class="wrap">
        <div id="container">
            <ul class="list">
                 <li class="item-0">first item</li>
                 <li class="item-1"><a href="link2.html">second item</a></li>
                 <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>
                 <li class="item-1 active"><a href="link4.html">fourth item</a></li>
                 <li class="item-0"><a href="link5.html">fifth item</a></li>
             </ul>
        </div>
    </div>
    """
    from pyquery import PyQuery as pq
    d = pq(html)
    li = d(".item-0.active")
    print(li)
    li.attr("name", "link")
    print(li)
    li.css("font-size", "14px")
    print(li)
    """
    <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>        
    <li class="item-0 active" name="link"><a href="link3.html"><span class="bold">third item</span></a></li>       
    <li class="item-0 active" name="link" style="font-size: 14px"><a href="link3.html"><span class="bold">third item</span></a></li>
    """

### remove() ###

![ContractedBlock.gif][] ![ExpandedBlockStart.gif][]

html = """
    <div class="wrap">
        Hello, World.
        <p>This is a paragraph.</p>
     </div>
    """
    
    from pyquery import PyQuery as pq
    d = pq(html)
    wrap = d(".wrap")
    print(wrap.text())
    """
    Hello, World.
    This is a paragraph.
    """
    wrap.find("p").remove()
    print(wrap.text())  # Hello, World.

### 其他DOM方法 ###

[https://pyquery.readthedocs.io/en/latest/api.html][https_pyquery.readthedocs.io_en_latest_api.html]

## 七、伪类选择器 ##

![ContractedBlock.gif][] ![ExpandedBlockStart.gif][]

html = """
    <div class="wrap">
        <div id="container">
            <ul class="list">
                 <li class="item-0">first item</li>
                 <li class="item-1"><a href="link2.html">second item</a></li>
                 <li class="item-0 active"><a href="link3.html"><span class="bold">third item</span></a></li>
                 <li class="item-1 active"><a href="link4.html">fourth item</a></li>
                 <li class="item-0"><a href="link5.html">fifth item</a></li>
            </ul>
        </div>
    </div>
    """
    from pyquery import PyQuery as pq
    d = pq(html)
    li = d("li:first-child")
    print(li)  # <li class="item-0">first item</li>
    li = d("li:last-child")
    print(li)  # <li class="item-0"><a href="link5.html">fifth item</a></li>
    li = d("li:nth-child(2)")
    print(li)  # <li class="item-1"><a href="link2.html">second item</a></li>
    li = d("li:gt(2)")  # 从0开始计数，索引大于2
    print(li)
    """
    <li class="item-1 active"><a href="link4.html">fourth item</a></li>
     <li class="item-0"><a href="link5.html">fifth item</a></li>
    """
    li = d("li:nth-child(2n)")  # 获取偶数顺序的元素（从1开始）
    print(li)
    """
    <li class="item-1"><a href="link2.html">second item</a></li>
    <li class="item-1 active"><a href="link4.html">fourth item</a></li>
    """
    li = d("li:contains(second)")  # 根据文本匹配，匹配文本包含second的标签
    print(li)  # <li class="item-1"><a href="link2.html">second item</a></li>

更多选择器：[http://www.w3school.com.cn/cssref/css\_selectors.asp][http_www.w3school.com.cn_cssref_css_selectors.asp]

转载于:https://www.cnblogs.com/believepd/p/10657877.html

[https_pyquery.readthedocs.io_en_latest]: https://pyquery.readthedocs.io/en/latest/
[ContractedBlock.gif]: https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif
[ExpandedBlockStart.gif]: /images/20220109/5c009a561ec747ff9866b67419e711ba.png
[https_pyquery.readthedocs.io_en_latest_api.html]: https://pyquery.readthedocs.io/en/latest/api.html
[http_www.w3school.com.cn_cssref_css_selectors.asp]: http://www.w3school.com.cn/cssref/css_selectors.asp

发表评论取消回复

表情：

评论列表（有 0 条评论，150人围观）

还没有评论，来说两句吧...

相关阅读

相关 PyQuery 库

强大又灵活的网页解析库。如果你觉得正则写起来太麻烦，BearutifulSoup 语法太难记，而又熟悉 jQuery 的语法，那么 PyQuery 就是你的绝佳选择 1、初

待我称王封你为后i/ 2023年10月17日 22:57/ 0 赞/ 57 阅读

相关如何使用PyQuery库？

PyQuery是一个类似于jQuery的Python库，它提供了一种可用于解析和操作HTML文档的强大工具。如何使用PyQuery库呢，下面是使用PyQuery库的详细说明：

短命女/ 2023年10月13日 20:18/ 0 赞/ 43 阅读

相关爬虫笔记：pyquery详解

pyquery 强大又灵活的网页解析库，如果你觉得正则写起来太麻烦，如果你觉得BeautifuiSoup语法太难记，如果你熟悉JQuery的语法，那么PyQuery就是你

ゞ浴缸里的玫瑰/ 2023年01月12日 10:09/ 0 赞/ 186 阅读

相关 19-爬虫解析利器pyquery详解

强大灵活的网页解析库。如果你觉得正则写起来太麻烦，或者BeautifulSoup语法太难记，如果你熟悉jQuery的语法，那么pyquery就是最佳选择。它与jQuery ap

喜欢ヅ旅行/ 2022年12月15日 03:47/ 0 赞/ 234 阅读

相关 PyQuery库使用详解。

转自：[http://www.pinlue.com/article/2020/09/0900/1211212911516.html][http_www.pinlue.com_a

今天药忘吃喽~/ 2022年12月06日 15:50/ 0 赞/ 149 阅读

相关爬虫之pyquery库

官方文档：[https://pyquery.readthedocs.io/en/latest/][https_pyquery.readthedocs.io_en_latest]

刺骨的言语ヽ痛彻心扉/ 2022年10月03日 00:48/ 0 赞/ 151 阅读

相关 Python爬虫-PyQuery

Python爬虫-PyQuery Python爬虫-PyQuery 一些看法使用语法

深碍√TFBOYSˉ_/ 2022年05月22日 08:27/ 0 赞/ 210 阅读

相关 python爬虫网页解析之pyquery的详解

一，写在前面 1，前文已经进行了四种网页解析工具的对比，现在我们来具体介绍一下pyquery的使用； 2，本文的参考资料来源于pyquery的[官方文档][Link 1]，

骑猪看日落/ 2022年05月14日 03:26/ 0 赞/ 222 阅读

相关【零基础学爬虫】PyQuery详解

回顾之前介绍了Beautifulsoup库，这个库可以让我们不写繁杂的正则表达式就可以爬取数据。但是你可能会觉得Beautifulsoup库不太好用，语法太繁杂，难记。

拼搏现实的明天。/ 2022年04月24日 07:22/ 0 赞/ 428 阅读

相关爬虫爬取小说（pyquery）

from pyquery import PyQuery import requests def crawle(): url = 'htt

本是古典何须时尚/ 2021年12月13日 15:04/ 0 赞/ 424 阅读