scrapy 安装出错 centos6 requirement Twisted>=13.1.0

ゞ浴缸里的玫瑰 2022-06-17 01:37 196阅读 0赞

Wednesday night： redhat/centos6.7 系统 安装scrapy时，提示缺少 twisted。然后用pip install twisted也提示找不到合适版本。

最终下载twisted源码安装，再pip install scrapy。问题就解决了。

$ vim make_scrapy.sh
    $ cat make_scrapy.sh

# virtualenv --python=python3.6 ENV
    # source ENV/bin/activate
    wget https://twistedmatrix.com/Releases/Twisted/17.1/Twisted-17.1.0.tar.bz2
    tar -jxvf Twisted-17.1.0.tar.bz2
    cd Twisted-17.1.0
    python setup.py install
    cd ..
    pip install scrapy

$ chmod +x make_scrapy.sh
    $ ./make_scrapy.sh

详情如下：

\[willie@bogon spiders\]$ django-admin startproject beautygirls

\[willie@bogon spiders\]$ cd beautygirls

\[willie@bogon beautygirls\]$ vvv   \#在~/.bashrc文件中 alias vvv=‘virtualenv --python=python3 venv'  
Running virtualenv with interpreter /usr/local/bin/python3  
Using base prefix '/opt/Python352'  
New python executable in /home/willie/projects/spiders/beautygirls/venv/bin/python3  
Also creating executable in /home/willie/projects/spiders/beautygirls/venv/bin/python  
Installing setuptools, pip, wheel...done.  
\[willie@bogon beautygirls\]$ ve   \#在~/.bashrc文件中 alias ve="source venv/bin/activate"  
(venv) \[willie@bogon beautygirls\]$ pip install scrapy  
Collecting scrapy  
  Using cached Scrapy-1.3.3-py2.py3-none-any.whl  
Collecting PyDispatcher>=2.0.5 (from scrapy)  
Collecting cssselect>=0.9 (from scrapy)  
  Using cached cssselect-1.0.1-py2.py3-none-any.whl  
Collecting parsel>=1.1 (from scrapy)  
  Using cached parsel-1.1.0-py2.py3-none-any.whl  
Collecting pyOpenSSL (from scrapy)  
  Using cached pyOpenSSL-16.2.0-py2.py3-none-any.whl  
Collecting queuelib (from scrapy)  
  Using cached queuelib-1.4.2-py2.py3-none-any.whl  
Collecting w3lib>=1.15.0 (from scrapy)  
  Using cached w3lib-1.17.0-py2.py3-none-any.whl  
Collecting lxml (from scrapy)  
  Using cached lxml-3.7.3-cp35-cp35m-manylinux1\_x86\_64.whl  
Collecting service-identity (from scrapy)  
  Using cached service\_identity-16.0.0-py2.py3-none-any.whl  
Collecting Twisted>=13.1.0 (from scrapy)  
  Could not find a version that satisfies the requirement Twisted>=13.1.0 (from scrapy) (from versions: )

No matching distribution found for Twisted>=13.1.0 (from scrapy)

\-------------------------------------------------------------------------------------------------

(venv) \[willie@bogon beautygirls\]$ pip install Twisted  
Collecting Twisted  
  Could not find a version that satisfies the requirement Twisted (from versions: )  
No matching distribution found for Twisted  
(venv) \[willie@bogon beautygirls\]$ pip install twisted  
Collecting twisted  
  Could not find a version that satisfies the requirement twisted (from versions: )  
No matching distribution found for twisted

\--------------------------------------------------------------------------------------------------

(venv) \[willie@bogon beautygirls\]$ pip install --verbose twisted

...

Traceback (most recent call last):  
  File "/home/willie/projects/spiders/beautygirls/venv/lib/python3.5/site-packages/pip/basecommand.py", line 215, in main  
    status = self.run(options, args)  
  File "/home/willie/projects/spiders/beautygirls/venv/lib/python3.5/site-packages/pip/commands/install.py", line 335, in run  
    wb.build(autobuilding=True)  
  File "/home/willie/projects/spiders/beautygirls/venv/lib/python3.5/site-packages/pip/wheel.py", line 749, in build  
    self.requirement\_set.prepare\_files(self.finder)  
  File "/home/willie/projects/spiders/beautygirls/venv/lib/python3.5/site-packages/pip/req/req\_set.py", line 380, in prepare\_files  
    ignore\_dependencies=self.ignore\_dependencies))  
  File "/home/willie/projects/spiders/beautygirls/venv/lib/python3.5/site-packages/pip/req/req\_set.py", line 554, in \_prepare\_file  
    require\_hashes  
  File "/home/willie/projects/spiders/beautygirls/venv/lib/python3.5/site-packages/pip/req/req\_install.py", line 278, in populate\_link  
    self.link = finder.find\_requirement(self, upgrade)  
  File "/home/willie/projects/spiders/beautygirls/venv/lib/python3.5/site-packages/pip/index.py", line 514, in find\_requirement  
    'No matching distribution found for %s' % req  
pip.exceptions.DistributionNotFound: No matching distribution found for twisted

\------------------------------------------------------------------------------------------------------

the way I fix it（写了个小脚本自动下载和安装twisted及scrapy）:

(venv) \[willie@bogon beautygirls\]$ vim make\_scrapy.sh  
(venv) \[willie@bogon beautygirls\]$ cat make\_scrapy.sh   
\#virtualenv --python=python3.5 venv  
\#source venv/bin/activate  
wget https://twistedmatrix.com/Releases/Twisted/17.1/Twisted-17.1.0.tar.bz2  
tar -jxvf Twisted-17.1.0.tar.bz2  
cd Twisted-17.1.0  
python setup.py install  
cd ..  
pip install scrapy

(venv) \[willie@bogon beautygirls\]$ source make\_scrapy.sh

.....wait for a few minutes 几分钟过后就自动安装完成了

。。。

Failed cleaning build dir for cryptography  
Failed to build cryptography  
Installing collected packages: cssselect, PyDispatcher, w3lib, idna, asn1crypto, pycparser, cffi, cryptography, pyOpenSSL, pyasn1, pyasn1-modules, service-identity, lxml, queuelib, parsel, scrapy  
  Running setup.py install for cryptography ... done  
Successfully installed PyDispatcher-2.0.5 asn1crypto-0.22.0 cffi-1.10.0 cryptography-1.8.1 cssselect-1.0.1 idna-2.5 lxml-3.7.3 parsel-1.1.0 pyOpenSSL-16.2.0 pyasn1-0.2.3 pyasn1-modules-0.0.8 pycparser-2.17 queuelib-1.4.2 scrapy-1.3.3 service-identity-16.0.0 w3lib-1.17.0  
(venv) \[willie@bogon beautygirls\]$ scrapy version  
Scrapy 1.3.3

That's all,well done!

安装完成，爬个美女图片庆祝一下吧。