selenium+webDriver+headless Chrome实现python爬虫

刺骨的言语ヽ痛彻心扉 2022-12-29 06:40 213阅读 0赞

设置浏览器的无界面状态可提高爬虫速度，在代码文件中进行设置即可。

from selenium import webdriver  
from selenium.webdriver.chrome.options import Options  
chrome\_options = Options()  
chrome\_options.add\_argument(’–headless’)  
chrome\_options.add\_argument(’–disable-gpu’)  
driver = webdriver.Chrome(chrome\_options=chrome\_options)  
编写代码

from selenium import webdriver  
import csv  
from selenium.webdriver.chrome.options import Options  
chrome\_options = Options()  
chrome\_options.add\_argument(’–headless’)  
chrome\_options.add\_argument(’–disable-gpu’)  
driver = webdriver.Chrome(chrome\_options=chrome\_options)  
url = ‘http://music.163.com/\#/discover/playlist/?order=hot&cat=%E5%85%A8%E9%83%A8&limit=35&offset=0’  
\#准备好存储歌单的csv文件  
csv\_file = open(“playlist.csv”,‘w’,newline=’’)  
writer = csv.writer(csv\_file)  
writer.writerow(\[‘标题’,‘播放数’,‘链接’\])  
\#解析每一页，直到‘下一页’为空  
while url != ‘javascript:void(0)’:  
\#用webDriver加载页面  
driver.get(url)  
\#切换到内容的iframe  
driver.switch\_to.frame(“contentFrame”)  
\#定位歌单标签  
data = driver.find\_element\_by\_id(“m-pl-container”).  
find\_elements\_by\_tag\_name(“li”)  
\#解析一页中的所有歌单  
for i in range(len(data)):  
\#获取播放数  
nb = data\[i\].find\_element\_by\_class\_name(“nb”).text  
if ‘万’ in nb and int(nb.split(“万”)\[0\]) > 500 :  
msk = data\[i\].find\_element\_by\_css\_selector(“a.msk”)  
writer.writerow(\[msk.get\_attribute(‘title’),nb,msk.get\_attribute(‘href’)\])  
url = driver.find\_element\_by\_css\_selector(“a.zbtn.znxt”).  
get\_attribute(“href”)  
csv\_file.close()  
运行结果  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xkcV9zZA_size_16_color_FFFFFF_t_70]

四、补充

在安装工具或者运行项目时经常遇到python版本不匹配的问题，这时我们可以进行版本切换，参考

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2xkcV9zZA_size_16_color_FFFFFF_t_70]: /images/20221120/0ff3651b1cb94c49982288a0820e1d11.png