爬虫笔记：Python Selenium详解

末蓝、 2022-10-29 11:17 221阅读 0赞

1.简介  
Selenium是一个用于测试网站的自动化测试工具，支持各种浏览器包括Chrome、Firefox、Safari等主流界面浏览器，同时也支持phantomJS无界面浏览器。  
支持多种操作系统如Windows、Linux、IOS、Android等。  
2.安装  
pip install Selenium  
3.安装浏览器驱动  
当selenium升级到3.0之后，对不同的浏览器驱动进行了规范。如果想使用selenium驱动不同的浏览器，必须单独下载并设置不同的浏览器驱动。

各浏览器下载地址：

Firefox浏览器驱动：[https://github.com/mozilla/geckodriver/releases][https_github.com_mozilla_geckodriver_releases]  
Chrome浏览器驱动：[链接1][1]  
[链接2][2]

IE浏览器驱动：[http://selenium-release.storage.googleapis.com/index.html][http_selenium-release.storage.googleapis.com_index.html]  
Edge浏览器驱动：[https://developer.microsoft.com/en-us/microsoft-edge/tools/webdriver/][https_developer.microsoft.com_en-us_microsoft-edge_tools_webdriver]  
Opera浏览器驱动：[https://github.com/operasoftware/operachromiumdriver/releases][https_github.com_operasoftware_operachromiumdriver_releases]  
PhantomJS浏览器驱动：[https://phantomjs.org/][https_phantomjs.org]

1.基本使用

from selenium import webdriver#浏览器驱动对象
    from selenium.webdriver.common.by import By
    from selenium.webdriver.common.keys import Keys
    from selenium.webdriver.support import expected_conditions as EC
    from selenium.webdriver.support.wait import WebDriverWait
    
    path='D:\chromedriver_win32\chromedriver.exe'#驱动目录
    browser = webdriver.Chrome(path)#声明一个浏览器对象
    try:
        browser.get('https://www.baidu.com')#访问网址
        input = browser.find_element_by_id('kw')
        input.send_keys('Python')#在键盘里输入python
        input.send_keys(Keys.ENTER)#输入回车
        wait = WebDriverWait(browser, 10)#等待10秒
        wait.until(EC.presence_of_element_located((By.ID, 'content_left')))#等待ID为content_left加载出来
        print(browser.current_url)
        print(browser.get_cookies())
        print(browser.page_source)#源代码
    finally:
        browser.close()

![20210213092923285.png_pic_center][]

2.声明浏览器对象

from selenium import webdriver
    
    browser = webdriver.Chrome()
    browser = webdriver.Firefox()
    browser = webdriver.Edge()
    browser = webdriver.PhantomJS()
    browser = webdriver.Safari()

3.访问页面

from selenium import webdriver
    path='D:\chromedriver_win32\chromedriver.exe'#驱动目录
    browser = webdriver.Chrome(path)
    browser.get('https://www.taobao.com')
    print(browser.page_source)#打印源代码
    #browser.close()

浏览器会自动打开淘宝页面，如果要关闭，可以在末尾添加browser.close()。  
浏览器淘宝页面会一闪而过  
![20210213093620452.png_pic_center][]  
page\_source  
![20210213095420974.png_pic_center][]

**查找元素**  
4.查找单个元素

from selenium import webdriver
    path='D:\chromedriver_win32\chromedriver.exe'#驱动目录
    browser = webdriver.Chrome(path)
    browser.get('https://www.taobao.com')
    input_first = browser.find_element_by_id('q')
    input_second = browser.find_element_by_css_selector('#q')
    input_third = browser.find_element_by_xpath('//*[@id="q"]')
    print(input_first, input_second, input_third)
    browser.close()

![20210213095752886.png_pic_center][]

定位元素方法  
![20210213100022621.png_pic_center][]

通用方法

from selenium import webdriver
    from selenium.webdriver.common.by import By
    path='D:\chromedriver_win32\chromedriver.exe'#驱动目录
    browser = webdriver.Chrome(path)
    browser.get('https://www.taobao.com')
    input_first = browser.find_element(By.ID, 'q')#通过id
    print(input_first)
    browser.close()

5.查找多个元素

from selenium import webdriver
    path='D:\chromedriver_win32\chromedriver.exe'#驱动目录
    browser = webdriver.Chrome(path)
    browser.get('https://www.taobao.com')
    lis = browser.find_elements_by_css_selector('.service-bd li')
    print(lis)
    browser.close()

![20210213100447996.png_pic_center][] 选中的是左侧导航栏标签 ![20210213100827562.png_pic_center][]

通用写法

from selenium import webdriver
    from selenium.webdriver.common.by import By
    
    path='D:\chromedriver_win32\chromedriver.exe'#驱动目录
    browser = webdriver.Chrome(path)
    browser.get('https://www.taobao.com')
    lis = browser.find_elements(By.CSS_SELECTOR, '.service-bd li')
    print(lis)
    browser.close()

**元素交互操作**  
6.对获取的元素调用交互方法

from selenium import webdriver
    import time
    
    path='D:\chromedriver_win32\chromedriver.exe'#驱动目录
    browser = webdriver.Chrome(path)
    browser.get('https://www.taobao.com')
    input = browser.find_element_by_id('q')#捕捉输入框
    input.send_keys('iPhone')#在输入框里输入iphone
    time.sleep(1)#等待1秒
    input.clear()#清除
    input.send_keys('iPad')#再次输入ipad
    button = browser.find_element_by_class_name('btn-search')#搜索按钮
    button.click()#点击按钮

结果如图  
![20210213103157750.gif_pic_center][]

分析输入框id为q  
![20210213143454267.png_pic_center][]  
点击按钮的class为btn-search  
![20210213143832229.png_pic_center][]

更多操作：[http://selenium-python.readthedocs.io/api.html\#module-selenium.webdriver.remote.webelement][http_selenium-python.readthedocs.io_api.html_module-selenium.webdriver.remote.webelement]

**交互操作**  
7.交互动作,将动作附加到动作链中串行执行

from selenium import webdriver
    from selenium.webdriver import ActionChains
    path='D:\chromedriver_win32\chromedriver.exe'#驱动目录
    browser = webdriver.Chrome(path)
    url = 'http://www.runoob.com/try/try.php?filename=jqueryui-api-droppable'
    browser.get(url)
    browser.switch_to.frame('iframeResult')
    source = browser.find_element_by_css_selector('#draggable')
    target = browser.find_element_by_css_selector('#droppable')
    actions = ActionChains(browser)
    actions.drag_and_drop(source, target)
    actions.perform()

[更多操作][Link 1]

**执行JavaScript**

![20210213150321887.gif_pic_center][]

**获取元素信息**  
9.获取属性

from selenium import webdriver
    path='D:\chromedriver_win32\chromedriver.exe'#驱动目录
    
    from selenium.webdriver import ActionChains
    
    browser = webdriver.Chrome(path)
    url = 'https://blog.csdn.net/KOBEYU652453/article/details/113743933'
    browser.get(url)
    logo = browser.find_element_by_id('article_content')#定位
    print(logo)
    print(logo.get_attribute('class'))#获取logo的class

结果  
![20210213151530115.png][]

网页中的class  
![20210213151613498.png][]

10.获取文本值

from selenium import webdriver
    path='D:\chromedriver_win32\chromedriver.exe'#驱动目录
    
    browser = webdriver.Chrome(path)
    url = 'https://blog.csdn.net/KOBEYU652453/article/details/113743933'
    browser.get(url)
    
    input = browser.find_element_by_class_name('article_content')
    print(input.text)

![20210213162557980.png][]

11.获取ID、位置、标签名、大小

![20210213162942371.png][]

**Frame**

path='D:\chromedriver_win32\chromedriver.exe'#驱动目录
    
    import time
    from selenium import webdriver
    from selenium.common.exceptions import NoSuchElementException
    
    browser = webdriver.Chrome(path)
    url = 'http://www.runoob.com/try/try.php?filename=jqueryui-api-droppable'
    browser.get(url)
    browser.switch_to.frame('iframeResult')
    source = browser.find_element_by_css_selector('#draggable')
    print(source)
    try:
        logo = browser.find_element_by_class_name('logo')
    except NoSuchElementException:
        print('NO LOGO')
    browser.switch_to.parent_frame()
    logo = browser.find_element_by_class_name('logo')
    print(logo)
    print(logo.text)

![20210213163406356.png][]

**等待**

当使用了隐式等待执行测试的时候，如果 WebDriver没有在 DOM中找到元素，将继续等待，超出设定时间后则抛出找不到元素的异常, 换句话说，当查找元素或元素并没有立即出现的时候，隐式等待将等待一段时间再查找 DOM，默认的时间是0

from selenium import webdriver
    path='D:\chromedriver_win32\chromedriver.exe'#驱动目录
    
    browser = webdriver.Chrome(path)
    
    browser.implicitly_wait(10)
    browser.get('https://www.zhihu.com/explore')
    input = browser.find_element_by_class_name('zu-top-add-question')
    print(input)

**显示等待**

path='D:\chromedriver_win32\chromedriver.exe'#驱动目录
    from selenium import webdriver
    from selenium.webdriver.common.by import By
    from selenium.webdriver.support.ui import WebDriverWait
    from selenium.webdriver.support import expected_conditions as EC
    
    browser = webdriver.Chrome(path)
    browser.get('https://www.taobao.com/')
    wait = WebDriverWait(browser, 10)
    input = wait.until(EC.presence_of_element_located((By.ID, 'q')))#等待条件
    button = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, '.btn-search')))
    print(input, button)

等待条件如下

* title_is 标题是某内容
    * title_contains  标题包含某内容
    * presence_of_element_located 元素加载出，传入定位元组，如(By.ID, 'p')
    * visibility_of_element_located  元素可见，传入定位元组
    * visibility_of 可见，传入元素对象
    * presence_of_all_elements_located 所有元素加载出
    * text_to_be_present_in_element 某个元素文本包含某文字
    * text_to_be_present_in_element_value  某个元素值包含某文字
    * frame_to_be_available_and_switch_to_it  frame加载并切换
    * invisibility_of_element_located 元素不可见
    * element_to_be_clickable  元素可点击
    * staleness_of  判断一个元素是否仍在DOM，可判断页面是否已经刷新
    * element_to_be_selected  元素可选择，传元素对象
    * element_located_to_be_selected  元素可选择，传入定位元组
    * element_selection_state_to_be  传入元素对象以及状态，相等返回True，否则返回False
    * element_located_selection_state_to_be  传入定位元组以及状态，相等返回True，否则返回False
    * alert_is_present 是否出现Alert

[等待详细内容][Link 2]

**前进后退**

import time
    from selenium import webdriver
    path='D:\chromedriver_win32\chromedriver.exe'#驱动目录
    browser = webdriver.Chrome(path)
    browser.get('https://www.baidu.com/')
    browser.get('https://www.taobao.com/')
    browser.get('https://www.python.org/')
    browser.back()
    time.sleep(1)
    browser.forward()
    browser.close()

![20210213165702867.gif_pic_center][]

**Cookies**

from selenium import webdriver
    path='D:\chromedriver_win32\chromedriver.exe'#驱动目录
    browser = webdriver.Chrome(path)
    browser.get('https://www.zhihu.com/explore')
    print(browser.get_cookies())#打印cookies
    browser.add_cookie({ 'name': 'name', 'domain': 'www.zhihu.com', 'value': 'germey'})#添加cookies
    print(browser.get_cookies())
    browser.delete_all_cookies()#删除所有cookies
    print(browser.get_cookies())

![20210213170824259.png_pic_center][]

**选项卡管理**

import time
    from selenium import webdriver
    path='D:\chromedriver_win32\chromedriver.exe'#驱动目录
    browser = webdriver.Chrome(path)
    browser.get('https://www.baidu.com')
    browser.execute_script('window.open()')
    print(browser.window_handles)
    browser.switch_to_window(browser.window_handles[1])
    browser.get('https://www.taobao.com')
    time.sleep(1)
    browser.switch_to_window(browser.window_handles[0])
    browser.get('https://python.org')

**异常处理**

path='D:\chromedriver_win32\chromedriver.exe'#驱动目录
    from selenium import webdriver
    from selenium.common.exceptions import TimeoutException, NoSuchElementException
    
    browser = webdriver.Chrome(path)
    try:
        browser.get('https://www.baidu.com')
    except TimeoutException:
        print('Time Out')
    try:
        browser.find_element_by_id('hello')
    except NoSuchElementException:
        print('No Element')
    finally:
        browser.close()

详细文档  
[http://selenium-python.readthedocs.io/api.html\#module-selenium.common.exceptions][http_selenium-python.readthedocs.io_api.html_module-selenium.common.exceptions]

![在这里插入图片描述][20210213172048767.jpg_pic_center]  
![在这里插入图片描述][2021021317205747.gif_pic_center]

[https_github.com_mozilla_geckodriver_releases]: https://github.com/mozilla/geckodriver/releases
[1]: https://sites.google.com/a/chromium.org/chromedriver/home
[2]: https://chromedriver.storage.googleapis.com/index.html?path=2.35/
[http_selenium-release.storage.googleapis.com_index.html]: http://selenium-release.storage.googleapis.com/index.html
[https_developer.microsoft.com_en-us_microsoft-edge_tools_webdriver]: https://developer.microsoft.com/en-us/microsoft-edge/tools/webdriver/
[https_github.com_operasoftware_operachromiumdriver_releases]: https://github.com/operasoftware/operachromiumdriver/releases
[https_phantomjs.org]: https://phantomjs.org/
[20210213092923285.png_pic_center]: /images/20221024/8ab56dfddd82451b9290ea7648be2071.png
[20210213093620452.png_pic_center]: /images/20221024/5f691c7873424e8daaee2fb99d27ff72.png
[20210213095420974.png_pic_center]: /images/20221024/55a12de3683546df980e11f2241832c2.png
[20210213095752886.png_pic_center]: /images/20221024/1638b2f0e594430394395970c6ae5f50.png
[20210213100022621.png_pic_center]: /images/20221024/2fecb5482e51493688f92b0596640a55.png
[20210213100447996.png_pic_center]: https://img-blog.csdnimg.cn/20210213100447996.png#pic_center
[20210213100827562.png_pic_center]: /images/20221024/96c02ac5cfc34ea09785816beb4b351c.png
[20210213103157750.gif_pic_center]: /images/20221024/db59167171784e0da736cb6974269ce8.png
[20210213143454267.png_pic_center]: /images/20221024/cd2254bd27c74f55b0c37d48542a8654.png
[20210213143832229.png_pic_center]: /images/20221024/638dc07e1db54a28a287acedb820c5cb.png
[http_selenium-python.readthedocs.io_api.html_module-selenium.webdriver.remote.webelement]: http://selenium-python.readthedocs.io/api.html#module-selenium.webdriver.remote.webelement
[Link 1]: http://selenium-python.readthedocs.io/api.html#module-selenium.webdriver.common.action_chains
[20210213150321887.gif_pic_center]: /images/20221024/3c85ed7024ae4519b92226242f869581.png
[20210213151530115.png]: /images/20221024/00d3e889200247c2a836e3a5ba68eb66.png
[20210213151613498.png]: /images/20221024/0b467c96e27b4c7b890fa6ef8dfee927.png
[20210213162557980.png]: /images/20221024/f2f4d5730cc0434fa5eac972b3ddbdcb.png
[20210213162942371.png]: /images/20221024/b7b0543bcfbe4b408dd9c9f2f60529c6.png
[20210213163406356.png]: /images/20221024/f7d68e9ddf3d403fa94f967a7b61077b.png
[Link 2]: http://selenium-python.readthedocs.io/api.html#module-selenium.webdriver.support.expected_conditions
[20210213165702867.gif_pic_center]: /images/20221024/df23c7e270e14a2293ada219e4c42a63.png
[20210213170824259.png_pic_center]: /images/20221024/053210e7c669428c81df806becd3f24c.png
[http_selenium-python.readthedocs.io_api.html_module-selenium.common.exceptions]: http://selenium-python.readthedocs.io/api.html#module-selenium.common.exceptions
[20210213172048767.jpg_pic_center]: /images/20221024/c199a8e7841c414d8719b6381a15c92b.png
[2021021317205747.gif_pic_center]: /images/20221024/6aff1394cf2f46d3bbe32f67cf0e7830.png