【爬虫】lxml的使用（xpath选择器、获取文本内容、获取属性、string(.)）

迈不过友情╰ 2022-11-21 14:45 388阅读 0赞

#### 1.lxml ####

lxml是一个高效率的HTML或XML文档解析库，用C语言写的，所以解析速度很快，我们可以配合xpath语法使用

##### 1.1 xpath语法 #####

[【爬虫】元素定位（xpath、css）][xpath_css]

##### 1.2 常见的解析库 #####

<table> 
 <thead> 
  <tr> 
   <th>解析器</th> 
   <th>使用方法</th> 
   <th>优点</th> 
   <th>缺点</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>Python标准库</td> 
   <td>BeautifulSoup(markup, “html.parser”)</td> 
   <td>Python的内置标准库<br>执行速度适中<br>文档容错能力强</td> 
   <td>版本容错能力差</td> 
  </tr> 
  <tr> 
   <td>lxml HTML 解析器</td> 
   <td>BeautifulSoup(markup, “lxml”)</td> 
   <td>速度快<br>文档容错能力强</td> 
   <td>需要安装C语言库</td> 
  </tr> 
  <tr> 
   <td>lxml XML 解析器</td> 
   <td>BeautifulSoup(markup, [“lxml-xml”])<br> BeautifulSoup(markup, “xml”)</td> 
   <td>速度快<br>唯一支持XML的解析器</td> 
   <td>需要安装C语言库</td> 
  </tr> 
  <tr> 
   <td>html5lib</td> 
   <td>BeautifulSoup(markup, “html5lib”)</td> 
   <td>最好的容错性<br>以浏览器的方式解析文档<br>生成HTML5格式的文档</td> 
   <td>速度慢<br>不依赖外部扩展</td> 
  </tr> 
 </tbody> 
</table>

##### 1.3 安装lxml #####

pip install lxml

#### 2.简单使用 ####

from lxml import etree
    
    html = """ <html><head><title>The Dormouse's story</title></head> <body> <p class="title"><b>The Dormouse's story</b></p> <p class="story">Once upon a time there were three little sisters; and their names were <a href="http://example.com/elsie" class="sister" id="link1">Elsie</a>, <a href="http://example.com/lacie" class="sister" id="link2">Lacie</a> and <a href="http://example.com/tillie" class="sister" id="link3">Tillie</a>; <a class='sister'>哈哈哈</a> and they lived at the bottom of a well. </p> <p class="story">...</p> """
    
    # html = etree.parse('hello.html') # 读取本地文件
    
    html = etree.HTML(html)
    # text = etree.tostring(html, encoding="utf-8").decode() # 打印HTML源码
    # print(text)
    
    a_text = html.xpath("//a[@class='sister']/text()")  # 获取文本内容
    print("a_text", a_text)
    
    a_text = html.xpath("//p[@class='story']/text()")  # 获取文本内容，返回列表
    print("a_text: ", a_text)
    a_text = html.xpath("//p[@class='story']")[0].xpath("string(.)")  # 获取文本内容，不带列表的字符串，如果存在标签嵌套时可以使用
    print("a_text: ", a_text)
    
    
    a_href = html.xpath('//a[@id="link2"]/@href')  # 获取某个属性
    print("a_href: ", a_href)
    
    # 如果获得的是element对象，可以继续使用xpath
    a_element = html.xpath("//a[@class='sister']")
    print(a_element)
    a_list = []
    for a in a_element:
        item = dict()
        # 如果取不到就设置为None
        item["href"] = a.xpath('./@href')[0] if len(a.xpath('./@href')) > 0 else None
        item["text"] = a.xpath('./text()')[0] if len(a.xpath('./text()')) > 0 else None
        a_list.append(item)
    print(a_list)

#### 3.实际运用 ####

可查看之前的文章

[【python】爬取正在上映的豆瓣电影信息（etree、xpath）][python_etree_xpath]

[【python】爬取豆瓣电影Top250（etree、xpath）][python_Top250_etree_xpath]

[xpath_css]: https://blog.csdn.net/qq_39147299/article/details/109263051
[python_etree_xpath]: https://blog.csdn.net/qq_39147299/article/details/103644680
[python_Top250_etree_xpath]: https://blog.csdn.net/qq_39147299/article/details/103646383