爬虫学习笔记（三）requests模块使用

港控/mmm° 2022-05-20 00:27 311阅读 0赞

前面在说爬虫原理的时候说了，就写代码自动化的获取数据，保存下来数据，那怎么写代码来请求一个网址，获取结果呢，就得用requests模块了。

这篇博客说一下requests模块的使用，requests模块是python的一个第三方模块，它是基于python自带的urllib模块封装的，用来发送http请求和获取返回的结果，操作很简单。

requests模块是第三方模块，不是python自带的，需要安装才可以使用。

<table> 
 <tbody> 
  <tr> 
   <td> 
    <div style="font-size:12px;line-height:15px;"> 
     <div>
       1 
     </div> 
    </div> </td> 
   <td> 
    <div style="font-size:12px;line-height:15px;"> 
     <div> 
      <span>pip <span>install <span>requests</span></span></span> 
     </div> 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

### requests模块用法 ###

<table> 
 <tbody> 
  <tr> 
   <td> 
    <div style="font-size:12px;line-height:15px;"> 
     <div>
       1 
     </div> 
     <div>
       2 
     </div> 
     <div>
       3 
     </div> 
     <div>
       4 
     </div> 
     <div>
       5 
     </div> 
     <div>
       6 
     </div> 
     <div>
       7 
     </div> 
     <div>
       8 
     </div> 
     <div>
       9 
     </div> 
     <div>
       10 
     </div> 
     <div>
       11 
     </div> 
     <div>
       12 
     </div> 
     <div>
       13 
     </div> 
     <div>
       14 
     </div> 
     <div>
       15 
     </div> 
     <div>
       16 
     </div> 
     <div>
       17 
     </div> 
     <div>
       18 
     </div> 
     <div>
       19 
     </div> 
     <div>
       20 
     </div> 
     <div>
       21 
     </div> 
     <div>
       22 
     </div> 
     <div>
       23 
     </div> 
     <div>
       24 
     </div> 
     <div>
       25 
     </div> 
    </div> </td> 
   <td> 
    <div style="font-size:12px;line-height:15px;"> 
     <div> 
      <span>import<span> <span>requests</span></span></span> 
     </div> 
     <div>
       &nbsp; 
     </div> 
     <div> 
      <span>req<span> <span>=<span> <span>requests<span>.<span>get<span>(<span>'http://www.puuuq.cn'<span>,<span>data<span>=<span>{ <span>'username'<span>:<span>'xxx'<span>}<span>,<span>cookies<span>=<span>{ <span>'k'<span>:<span>'v'<span>}<span>,</span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <span>headers<span>=<span>{ <span>'User-Agent'<span>:<span>'Chrome'<span>}<span>,<span>verify<span>=<span>False<span>,<span>timeout<span>=<span>3<span>)<span>&nbsp;&nbsp;<span>#发送get请求，data是请求数据，</span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span># cookies是要发送的cookies，headers是请求头信息，verify=False是https请求的时候要加上，要不然会报错。</span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>#timeout参数是超时时间，超过几秒钟的话，就不再去请求它了，会返回timeout异常</span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>#这些都可以不写，如果有的话，可以加上</span></span> 
     </div> 
     <div> 
      <span>req2<span> <span>=<span> <span>requests<span>.<span>post<span>(<span>'http://www.puuuq.cn'<span>,<span>data<span>=<span>{ <span>'username'<span>:<span>'xxx'<span>}<span>,<span>cookies<span>=<span>{ <span>'k'<span>:<span>'v'<span>}<span>,</span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>headers<span>=<span>{ <span>'User-Agent'<span>:<span>'Chrome'<span>}<span>,<span>files<span>=<span>{ <span>'file'<span>:<span>open<span>(<span>'a.txt'<span>)<span>}<span>,<span>timeout<span>=<span>3<span>)<span> <span>#发送post请求，data是请求数据，</span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span># cookies是要发送的cookies，headers是请求头信息，files是发送的文件，verify=False是https请求的时候要加上，</span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span># 要不然会报错,timeout参数是超时时间，超过几秒钟的话，就不再去请求它了，会返回timeout异常</span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span>#这些都可以不写，如果有的话，可以加上</span></span> 
     </div> 
     <div>
       &nbsp; 
     </div> 
     <div> 
      <span>req3<span> <span>=<span> <span>requests<span>.<span>put<span>(<span>'http://www.puuuq.cn'<span>)<span> <span>#put方式请求</span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>req4<span> <span>=<span> <span>requests<span>.<span>patch<span>(<span>'http://www.puuuq.cn'<span>)<span>#patch方式请求</span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>req5<span> <span>=<span> <span>requests<span>.<span>delete<span>(<span>'http://www.puuuq.cn'<span>)<span>#delete方式请求</span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>req6<span> <span>=<span> <span>requests<span>.<span>options<span>(<span>'http://www.puuuq.cn'<span>)<span>#options方式请求，用法和上面的get、post都一样</span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div>
       &nbsp; 
     </div> 
     <div> 
      <span>print<span>(<span>req<span>.<span>status_code<span>)<span> <span>#获取返回状态码</span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>print<span>(<span>req<span>.<span>content<span>)<span>#获取返回的内容，二进制格式,一般下载图片、视频用这个</span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>print<span>(<span>req<span>.<span>text<span>)<span> <span>#获取返回的内容，字符串格式</span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>print<span>(<span>req<span>.<span>json<span>(<span>)<span>)<span>#获取返回的内容，json格式,这个必须是返回的是json才可以使用，否则会报错</span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>print<span>(<span>req<span>.<span>headers<span>)<span>#获取响应头</span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>print<span>(<span>req<span>.<span>cookies<span>)<span>#获取返回的cookie</span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>print<span>(<span>req<span>.<span>encoding<span>)<span>#获取返回的字符集</span></span></span></span></span></span></span> 
     </div> 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

### http权限认证 ###

有一些网站，比如说下载东西的时候有http的权限验证，没有验证话就返回401 请求未经授权这种错误的。一般都是需要http权限验证，下面是怎么添加http权限验证。

当然这个http权限认证是http本身的，和你那些登陆请求那些不一样，比如说你要调一个登陆接口，传入的账号密码，和那个不是一回事，要区别开。

举个例子呢就是商场的大门上的锁就是这个http权限验证，这个锁是人家商场的，而你的店铺的锁才是你登陆接口，你输入的账号密码。一般你一打开一个网站直接弹出来一个窗口让你输入账号密码，你都看不到页面，这种就是http权限验证。而那种你打开网站之后，直接就能看到页面，你要登录的时候，输入账号密码然后点登录的，这种的就是正常的登陆请求。这种http权限验证的比较少见。

<table> 
 <tbody> 
  <tr> 
   <td> 
    <div style="font-size:12px;line-height:15px;"> 
     <div>
       1 
     </div> 
     <div>
       2 
     </div> 
     <div>
       3 
     </div> 
     <div>
       4 
     </div> 
     <div>
       5 
     </div> 
     <div>
       6 
     </div> 
     <div>
       7 
     </div> 
     <div>
       8 
     </div> 
     <div>
       9 
     </div> 
     <div>
       10 
     </div> 
     <div>
       11 
     </div> 
    </div> </td> 
   <td> 
    <div style="font-size:12px;line-height:15px;"> 
     <div> 
      <span>import<span> <span>requests</span></span></span> 
     </div> 
     <div> 
      <span>from<span> <span>requests<span>.<span>auth <span>import<span> <span>HTTPBasicAuth</span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>#导入HTTPBasicAuth</span> 
     </div> 
     <div>
       &nbsp; 
     </div> 
     <div> 
      <span>req<span> <span>=<span> <span>requests<span>.<span>post<span>(<span>'http://www.puuuq.cn'<span>,<span>data<span>=<span>{ <span>'username'<span>:<span>'xxx'<span>}<span>,<span>auth<span>=<span>HTTPBasicAuth<span>(<span>'username'<span>,<span>'password'<span>)<span>)</span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>#使用的时候加上auth参数，然后使用HTTPBasicAuth，传入账号和密码即可。其他的都是和以前一样用</span> 
     </div> 
     <div> 
      <span>print<span>(<span>req<span>.<span>status_code<span>)</span></span></span></span></span></span> 
     </div> 
     <div>
       &nbsp; 
     </div> 
     <div>
       &nbsp; 
     </div> 
     <div>
       &nbsp; 
     </div> 
     <div>
       &nbsp; 
     </div> 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

### http会话保持 ###

什么是会话保持，就是有一些操作需要登录之后才操作的，你得先登录，然后才能做其他的操作。那咱们做的时候怎么做，先发送登陆的请求，获取到登录的cookie信息，（因为登录之后它的身份验证信息都是放在cookie里面的），然后把cookie传给下一个你要请求的url，这样就ok了，看下面代码。

<table> 
 <tbody> 
  <tr> 
   <td> 
    <div style="font-size:12px;line-height:15px;"> 
     <div>
       1 
     </div> 
     <div>
       2 
     </div> 
     <div>
       3 
     </div> 
     <div>
       4 
     </div> 
     <div>
       5 
     </div> 
     <div>
       6 
     </div> 
     <div>
       7 
     </div> 
     <div>
       8 
     </div> 
    </div> </td> 
   <td> 
    <div style="font-size:12px;line-height:15px;"> 
     <div> 
      <span>import<span> <span>requests</span></span></span> 
     </div> 
     <div> 
      <span>r1<span>=<span>requests<span>.<span>post<span>(<span>'http://www.puuuq.cn/login'<span>,<span>data<span>=<span>{ <span>'username'<span>:<span>'besttest'<span>,<span>'password'<span>:<span>'123456'<span>}<span>)<span>#登陆请求</span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>login_cookies<span> <span>=<span> <span>r1<span>.<span>cookies<span> <span>#获取到登陆请请求返回的cookie</span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>r2<span> <span>=<span> <span>requests<span>.<span>post<span>(<span>'http://www.puuuq/create_user'<span>,</span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <span>data<span>=<span>{ <span>'title'<span>:<span>'测试测试'<span>,<span>'content'<span>:<span>'发送文章测试'<span>}<span>,</span></span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <span>cookies<span>=<span>login_cookies<span>)</span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;<span>#把登陆获取到的cookie，传给发表文章的这个请求，就ok了</span></span> 
     </div> 
     <div> 
      <span>print<span>(<span>r2<span>.<span>text<span>)</span></span></span></span></span></span> 
     </div> 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

正常的话咱们要这么做，requests模块给咱们提供了更简单的方式，就是使用requests.session这个方法，它会自动帮咱们管理cookie，不需要咱们自己再获取到登陆的cookie，传给创建文件的请求，代码如下：

<table> 
 <tbody> 
  <tr> 
   <td> 
    <div style="font-size:12px;line-height:15px;"> 
     <div>
       1 
     </div> 
     <div>
       2 
     </div> 
     <div>
       3 
     </div> 
     <div>
       4 
     </div> 
     <div>
       5 
     </div> 
     <div>
       6 
     </div> 
     <div>
       7 
     </div> 
     <div>
       8 
     </div> 
    </div> </td> 
   <td> 
    <div style="font-size:12px;line-height:15px;"> 
     <div> 
      <span>import<span> <span>requests</span></span></span> 
     </div> 
     <div> 
      <span>r<span>=<span>requests<span>.<span>session<span>(<span>)</span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>login_req<span> <span>=<span> <span>r<span>.<span>post<span>(<span>'http://www.puuuq.cn/login'<span>,<span>data<span>=<span>{ <span>'username'<span>:<span>'besttest'<span>,<span>'password'<span>:<span>'123456'<span>}<span>)<span> <span>#发送登陆的请求</span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div>
       &nbsp; 
     </div> 
     <div> 
      <span>r1<span> <span>=<span> <span>r<span>.<span>post<span>(<span>'http://www.puuuq.cn/create_user'<span>,</span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div> 
      <span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; <span>data<span>=<span>{ <span>'title'<span>:<span>'测试测试'<span>,<span>'content'<span>:<span>'发送文章测试'<span>}<span>)<span> <span>#发送创建文件的请求</span></span></span></span></span></span></span></span></span></span></span></span></span></span></span> 
     </div> 
     <div>
       &nbsp; 
     </div> 
     <div> 
      <span>print<span>(<span>r1<span>.<span>text<span>)<span>#获取返回的结果</span></span></span></span></span></span></span> 
     </div> 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

### http代理设置 ###

我们在写爬虫的时候，如果都用同一个ip访问多次访问某个网站，经常ip会被封，这样我们就访问不了了，那怎么解决呢，就得用ip代理了，代理的意思就是咱们把请求先发到代理上，然后再由代理帮咱们把请求发送出去，这样最终访问网站的ip就不是咱们自己的ip了。网上代理有很多，大多数收费的代理网站上每天都会有几个免费的代理，我先从https://www.kuaidaili.com/free/inha/ 这个里面找了几个免费的代理，设置代理的代码如下