Scrapy第十一(②)篇：selenium4模拟器中间件-多线程并发

向右看齐 2024-04-08 08:51 87阅读 0赞

#### 1.安装依赖 ####

pip install scrapy-ajax-utils -i https://pypi.tuna.tsinghua.edu.cn/simple

源码地址：[GitHub - kingronjan/scrapy\_ajax\_utils: utils for crawl ajax page in scrapy project.][GitHub - kingronjan_scrapy_ajax_utils_ utils for crawl ajax page in scrapy project.]

#### 2.修改spider文件 ####

①为spider加上装饰器@selenium\_support

②将spider.Request()方法换成SeleniumRequest()【重写start\_requests方法】

import scrapy
    from scrapy.http import HtmlResponse
    from scrapy_ajax_utils import selenium_support, SeleniumRequest
    
    
    @selenium_support
    class TestspiderSpider(scrapy.Spider):
        name = 'testspider'
        allowed_domains = ['www.baidu.com']
        start_urls = ['https://www.baidu.com' for i in range(20)]
    
        def start_requests(self):
            for url in self.start_urls:
                yield SeleniumRequest(url, callback=self.parse, dont_filter=True)
    
        def parse(self, response: HtmlResponse):
            pass

#### 3.配置：settings.py（如果使用的是webdriver-manager，如何查看驱动地址：看下面第4点） ####

# 只支持两种浏览器驱动 chrome（默认） & firefox.
    SELENIUM_DRIVER_NAME = 'chrome'
    # 浏览器驱动地址（必填）
    SELENIUM_DRIVER_PATH = r'C:\Users\sunziwen\.wdm\drivers\chromedriver\win32\98.0.4758\chromedriver.exe'
    # 是否为无头浏览器
    SELENIUM_HEADLESS = False
    # 页面加载超时时间
    SELENIUM_DRIVER_PAGE_LOAD_TIMEOUT = 30
    # selenium的最小并发数
    SELENIUM_MIN_DRIVERS = 5
    # selenium的最大并发数
    SELENIUM_MAX_DRIVERS = 10

或者也可以直接赋值：

from selenium.webdriver.chrome.service import Service as ChromeService
    from webdriver_manager.chrome import ChromeDriverManager
    
    # 只支持两种浏览器驱动 chrome（默认） & firefox.
    SELENIUM_DRIVER_NAME = 'chrome'
    # 浏览器驱动地址（必填）
    # SELENIUM_DRIVER_PATH = r'C:\Users\sunziwen\.wdm\drivers\chromedriver\win32\98.0.4758\chromedriver.exe'
    SELENIUM_DRIVER_PATH = ChromeService(executable_path=ChromeDriverManager().install()).path
    # 是否为无头浏览器
    SELENIUM_HEADLESS = False
    # 页面加载超时时间
    SELENIUM_DRIVER_PAGE_LOAD_TIMEOUT = 30
    # selenium的最小并发数
    SELENIUM_MIN_DRIVERS = 4
    # selenium的最大并发数
    SELENIUM_MAX_DRIVERS = 5

#### 4.查看驱动地址： ####

from selenium.webdriver.chrome.service import Service as ChromeService
    from webdriver_manager.chrome import ChromeDriverManager
    
    
    if __name__ == '__main__':
        # 初始化驱动
        service = ChromeService(executable_path=ChromeDriverManager().install())
        print("驱动地址：",service.path)

#### 5.运行测试：![5a199a6cbe614b7a80ad6d982ce02c70.png][] ####

#### 6.操作浏览器驱动 ####

①编写操作中间件：middlewares.py

from selenium.webdriver.chrome.webdriver import WebDriver
    from scrapy_ajax_utils import SeleniumRequest
    
    class SeleniumMiddleware(object):
        def process_request(self: WebDriver, request: SeleniumRequest, spider):
            # self就是驱动实例
            # 访问网页
            self.get(request.url)
            # 全屏
            self.maximize_window()
            # 阻塞3秒
            time.sleep(3)
            # network记录，这里可以获得所有ajax请求的结果
            # logs = get_xhr_logs(chrome)
            # 动态加载后的网页
            html = self.page_source
            # 退出浏览器
            self.quit()
            return scrapy.http.HtmlResponse(url=request.url, body=html.encode('utf-8'), encoding='utf-8', request=request)

②启动中间件：settings.py

DOWNLOADER_MIDDLEWARES = {
        'testscrapy.middlewares.SeleniumMiddleware': 543,
    }  # 下载中间件

③修改spider文件，将SeleniumRequest()的handle参数指定为操作中间件。

![e89d8e6b396f4579bc54331e41ec84e2.png][]

7.断点测试

![059efda9822545cb9cda2756aa8e48ef.png][]

[GitHub - kingronjan_scrapy_ajax_utils_ utils for crawl ajax page in scrapy project.]: https://github.com/kingronjan/scrapy_ajax_utils
[5a199a6cbe614b7a80ad6d982ce02c70.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/08/2a63f0bd4a574294875f66d40cbe9b6d.png
[e89d8e6b396f4579bc54331e41ec84e2.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/08/6ce8d45f50c54ada929b79b796ece6d8.png
[059efda9822545cb9cda2756aa8e48ef.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/08/08330fdb64504d2197d4a1355a9ff14a.png