python爬虫防止IP被封的一些措施（伪造User-Agent ，在每次重复爬取之间设置一个随机时间间隔，伪造cookies ,使用代理）

╰半橙微兮° 2021-11-10 05:54 570阅读 0赞

转载于：转载地址

# 伪造User-Agent #

在请求头中把`User-Agent`设置成浏览器中的`User-Agent`，来伪造浏览器访问。比如：

send_headers = {
            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36",
            "Connection": "keep-alive",
            "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8",
            "Accept-Language": "zh-CN,zh;q=0.8"}
    
    
    
    
    r = requests.get(url,timeout=30,headers = send_headers)

还可以先收集多种浏览器的`User-Agent`，每次发起请求时随机从中选一个使用，可以进一步提高安全性：

import requests,random
    
    user_agents = ['Mozilla/5.0 (Windows NT 6.1; rv:2.0.1) Gecko/20100101 Firefox/4.0.1','Mozilla/5.0 (Windows; U; Windows NT 6.1; en-us) AppleWebKit/534.50 (KHTML, like Gecko) Version/5.1 Safari/534.50','Opera/9.80 (Windows NT 6.1; U; en) Presto/2.8.131 Version/11.11']
    
    def get_html(url):
        headers = {'User-Agent':random.choice(user_agents)}
        resp = requests.get(url,headers = headers)
        return resp.text

user_agent = [
           "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)",
           "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; AcooBrowser; .NET CLR 1.1.4322; .NET CLR 2.0.50727)",
           "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; Acoo Browser; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506)",
           "Mozilla/4.0 (compatible; MSIE 7.0; AOL 9.5; AOLBuild 4337.35; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)",
           "Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US)",
           "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Win64; x64; Trident/5.0; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET CLR 2.0.50727; Media Center PC 6.0)",
           "Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4.0; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET CLR 1.0.3705; .NET CLR 1.1.4322)",
           "Mozilla/4.0 (compatible; MSIE 7.0b; Windows NT 5.2; .NET CLR 1.1.4322; .NET CLR 2.0.50727; InfoPath.2; .NET CLR 3.0.04506.30)",
           "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN) AppleWebKit/523.15 (KHTML, like Gecko, Safari/419.3) Arora/0.3 (Change: 287 c9dfb30)",
           "Mozilla/5.0 (X11; U; Linux; en-US) AppleWebKit/527+ (KHTML, like Gecko, Safari/419.3) Arora/0.6",
           "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.2pre) Gecko/20070215 K-Ninja/2.1.1",
           "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.9) Gecko/20080705 Firefox/3.0 Kapiko/3.0",
           "Mozilla/5.0 (X11; Linux i686; U;) Gecko/20070322 Kazehakase/0.4.5",
           "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.8) Gecko Fedora/1.9.0.8-1.fc10 Kazehakase/0.5.6",
           "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11",
           "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_3) AppleWebKit/535.20 (KHTML, like Gecko) Chrome/19.0.1036.7 Safari/535.20",
           "Opera/9.80 (Macintosh; Intel Mac OS X 10.6.8; U; fr) Presto/2.9.168 Version/11.52"
    ]

把上面随机选择一个User-Agent的代码封装成一个函数：

import random
    def get_headers():
        '''
        随机获取一个headers
        '''
        user_agents =  ['Mozilla/5.0 (Windows NT 6.1; rv:2.0.1) Gecko/20100101 Firefox/4.0.1','Mozilla/5.0 (Windows; U; Windows NT 6.1; en-us) AppleWebKit/534.50 (KHTML, like Gecko) Version/5.1 Safari/534.50','Opera/9.80 (Windows NT 6.1; U; en) Presto/2.8.131 Version/11.11']
        headers = {'User-Agent':random.choice(user_agents)}
        return headers

注：一些常见浏览器的`User-Agent`可参见：[https://blog.csdn.net/qianxing111/article/details/79415857][https_blog.csdn.net_qianxing111_article_details_79415857]

--------------------

# 在每次重复爬取之间设置一个随机时间间隔：  #

# 比如：
    time.sleep(random.randint(0,3))  # 暂停0~3秒的整数秒，时间区间：[0,3]
    # 或：
    time.sleep(random.random())  # 暂停0~1秒，时间区间：[0,1)

--------------------

# 伪造cookies #

若从浏览器中可以正常访问一个页面，则可以将浏览器中的cookies复制过来使用，比如：

cookies = dict(uuid='b18f0e70-8705-470d-bc4b-09a8da617e15',UM_distinctid='15d188be71d50-013c49b12ec14a-3f73035d-100200-15d188be71ffd')
    resp = requests.get(url,cookies = cookies)

# 把浏览器的cookies字符串转成字典
    def cookies2dict(cookies):
        items = cookies.split(';')
        d = {}
        for item in items:
            kv = item.split('=',1)
            k = kv[0]
            v = kv[1]
            d[k] = v
        return d

--------------------

# 使用代理 #

可以换着用多个代理IP来进行访问，防止同一个IP发起过多请求而被封IP，比如：

proxies = {'http':'http://10.10.10.10:8765','https':'https://10.10.10.10:8765'}
    resp = requests.get(url,proxies = proxies)
    # 注：免费的代理IP可以在这个网站上获取：http://www.xicidaili.com/nn/

[https_blog.csdn.net_qianxing111_article_details_79415857]: https://blog.csdn.net/qianxing111/article/details/79415857