爬取知乎碰到的问题-----------------------3、关于url中出现sign的解决办法

ゞ浴缸里的玫瑰 2022-03-26 09:26 192阅读 0赞

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3poYW9fNTM1MjI2OQ_size_16_color_FFFFFF_t_70][]

2019-01-16 18:08:37 \[scrapy.core.scraper\] ERROR: Spider error processing <GET https://www.zhihu.com/signin?next=http%3A%2F%2Fwww.zhihu.com%2Fpeople%2Fbantys> (referer: https://zhihu.com/people/libragold/following)  
Traceback (most recent call last):  
  File "D:\\Python36\\lib\\site-packages\\scrapy\\utils\\defer.py", line 102, in iter\_errback  
    yield next(it)  
  File "D:\\Python36\\lib\\site-packages\\scrapy\\spidermiddlewares\\offsite.py", line 30, in process\_spider\_output  
    for x in result:  
  File "D:\\Python36\\lib\\site-packages\\scrapy\\spidermiddlewares\\referer.py", line 339, in <genexpr>  
    return (\_set\_referer(r) for r in result or ())  
  File "D:\\Python36\\lib\\site-packages\\scrapy\\spidermiddlewares\\urllength.py", line 37, in <genexpr>  
    return (r for r in result or () if \_filter(r))  
  File "D:\\Python36\\lib\\site-packages\\scrapy\\spidermiddlewares\\depth.py", line 58, in <genexpr>  
    return (r for r in result or () if \_filter(r))  
  File "G:\\zhihuspider\\zhihuscrapy\\spiders\\zhihulogintest.py", line 149, in parse\_people  
    userlinks = userlinks\['initialState'\]\['entities'\]\['users'\]\[zhihu\_id\]  
KeyError: 'signin?next=http%3A%2F%2Fwww.zhihu.com%2Fpeople%2Fbantys'

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3poYW9fNTM1MjI2OQ_size_16_color_FFFFFF_t_70 1][]

解决办法：

在试了几次之后发现这个链接url，放到知乎的页面会显示该用户已停用，找不到该用户，但是有时候你直接复制的id，也会出现这种问题，所以目前就是直接将访问不到的记录下来，放到文件做最后的处理。

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3poYW9fNTM1MjI2OQ_size_16_color_FFFFFF_t_70 2][]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3poYW9fNTM1MjI2OQ_size_16_color_FFFFFF_t_70]: /images/20220326/da2d25f8b8324fe099bfecf0458a65b8.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3poYW9fNTM1MjI2OQ_size_16_color_FFFFFF_t_70 1]: /images/20220326/f90d841bde964cb1843fe8935a4fd064.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3poYW9fNTM1MjI2OQ_size_16_color_FFFFFF_t_70 2]: /images/20220326/6e8b9ea626bf4821934738ca6ae3e7da.png