scrapy在window中的安装与使用初步入门

迷南。 2022-07-15 04:30 137阅读 0赞

下载并安装下面的过程比较顺利，就省略了。

安装python,版本3.5,注意勾上添加到环境变量，不然后面又要折腾。

安装ptvs for vs2015

安装ironPython

安装各种包时碰到的坑：主要是安装scrapy时碰到的。  
1.安装twisted出错：  
提示缺少vcvarsall.bat  
解决方法：  
主要是vs2015安装不完整，需要重新加载iso安装Visual C++语言即可；  
  
  
2.安装scrapy出错:  
提示缺少libxml2  
解决方法：  
进入以下地址  
http://www.lfd.uci.edu/~gohlke/pythonlibs/\#lxml  
找到自己的版本下载后执行如下cmd命令：  
pip install F:\\python3.5\\lxml-3.6.4-cp35-cp35m-win32.whl  
  
  
  
  
3.运行scrapy报错  
提示没有win32api  
解决方案：  
原因是缺少win32,到 http://sourceforge.net/projects/pywin32/files/  
找到对应的版本进行下载，直接安装即可

提示没有找到python3.5

解决方法：执行下面的脚本：

#
    # script to register Python 2.0 or later for use with win32all
    # and other extensions that require Python registry settings
    #
    # written by Joakim Loew for Secret Labs AB / PythonWare
    #
    # source:
    # http://www.pythonware.com/products/works/articles/regpy20.htm
    #
    # modified by Valentine Gogichashvili as described in http://www.mail-archive.com/distutils-sig@python.org/msg10512.html
    
    import sys
      
    from winreg import *
    
    # tweak as necessary
    version = sys.version[:3]
    installpath = sys.prefix
    
    regpath = "SOFTWARE\\Python\\Pythoncore\\%s\\" % (version)
    installkey = "InstallPath"
    pythonkey = "PythonPath"
    pythonpath = "%s;%s\\Lib\\;%s\\DLLs\\" % (
        installpath, installpath, installpath
    )
      
    def RegisterPy():
        try:
            reg = OpenKey(HKEY_CURRENT_USER, regpath)
        except EnvironmentError as e:
            try:
                reg = CreateKey(HKEY_CURRENT_USER, regpath)
                SetValue(reg, installkey, REG_SZ, installpath)
                SetValue(reg, pythonkey, REG_SZ, pythonpath)
                CloseKey(reg)
            except:
                print("*** Unable to register!")
                return
            print("--- Python", version, "is now registered!")
            return
        if (QueryValue(reg, installkey) == installpath and
            QueryValue(reg, pythonkey) == pythonpath):
            CloseKey(reg)
            print("=== Python", version, "is already registered!")
            return
        CloseKey(reg)
        print("*** Unable to register!")
        print("*** You probably have another Python installation!")
      
    if __name__ == "__main__":
        RegisterPy()

4.HelloWord  
  
  
进入存储代码的目录中，运行下列cmd命令:  
scrapy startproject tutorial  
  
  
在 tutorial/spiders 目录下的新建 dmoz\_spider.py 文件，内容为:

import scrapy
    
    
    class DmozSpider(scrapy.Spider):
        name = "dmoz"
        allowed_domains = ["dmoz.org"]
        start_urls = [
            "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",
            "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/"
        ]
    
    
        def parse(self, response):
            filename = response.url.split("/")[-2]
            with open(filename, 'wb') as f:
                f.write(response.body)

进入项目的根目录，执行下列cmd命令:  
scrapy crawl dmoz  
  
  
  
示例：带Rule的CrawlSpider样例

import scrapy
    from scrapy.contrib.spiders import CrawlSpider, Rule
    from scrapy.contrib.linkextractors import LinkExtractor
    
    
    class MySpider(CrawlSpider):
        name = 'example.com'
        allowed_domains = ['example.com']
        start_urls = ['http://www.example.com']
    
    
        rules = (
            # 提取匹配 'category.php' (但不匹配 'subsection.php') 的链接并跟进链接(没有callback意味着follow默认为True)
            Rule(LinkExtractor(allow=('category\.php', ), deny=('subsection\.php', ))),
    
    
            # 提取匹配 'item.php' 的链接并使用spider的parse_item方法进行分析
            Rule(LinkExtractor(allow=('item\.php', )), callback='parse_item'),
        )
    
    
        def parse_item(self, response):
            self.log('Hi, this is an item page! %s' % response.url)
    
    
            item = scrapy.Item()
            item['id'] = response.xpath('//td[@id="item_id"]/text()').re(r'ID: (\d+)')
            item['name'] = response.xpath('//td[@id="item_name"]/text()').extract()
            item['description'] = response.xpath('//td[@id="item_description"]/text()').extract()
            return item

导航：

### [老程序员的python快速学习之旅][python] ###

[python]: http://blog.csdn.net/manimanihome/article/details/53032964