python与matlab爬虫百度图片首页的图片(urllib.request)

朴灿烈づ我的快乐病毒、 2022-06-10 10:36 154阅读 0赞

### python的爬虫实践 ###

python版本：python 3.6.0，实践内容：爬去百度图片首页的所有图片保存在本地，并批量读取图片并排列组合成一张大图。内容涉及：正则表达式，反盗链等操作。需要说明的使用伪装的headers，否则可能会出现404 forbidden.下载图片使用的是urllib.request.urlretrieve函数，也可以使用save函数直接保存在本地。

import urllib.request
    import re
    url = "http://image.baidu.com/search/index?tn=baiduimage&ct=201326592&lm=-1&cl=2&ie=gbk&word=%B1%DA%D6%BD&fr=ala&ala=1&pos=0&alatpl=wallpaper&oriquery=%E5%A3%81%E7%BA%B8"
    headers={   
                'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.137 Safari/537.36 LBBROWSER'  
            } 
    req = urllib.request.Request(url=url, headers=headers)#伪装浏览器  
    page = urllib.request.urlopen(req)
    html = page.read()
    imgre = re.compile(r'"objURL":"http://img2.niutuku.com(.*?)\.jpg",')#网页源代码，正则表达式
    html=html.decode('utf-8','ignore')#python3
    imglist = re.findall(imgre,html)
    print (imglist)
    x=0
    for imgurl in imglist: 
                  urllib.request.urlretrieve('http://img2.niutuku.com'+imgurl+'.jpg','E:\\'+'%s.jpg' % x)
          x= x+1

将爬取的图片保存在一张图上。

import matplotlib.pyplot as plt
    from scipy import misc
    for i in range(1,11):
       str = "%s%d%s"% ("E:\\",i-1,".jpg")
       image = mpimg.imread(str)#图片读取
       image_new_size = misc.imresize(image,[260,280])#统一尺寸
       plt.subplot(2,5,i)
       plt.imshow(image_new_size,origin = 'lower')
       plt.axis('off')
    plt.show()
    plt.savefig('E:\\out.jpg')

![这里写图片描述][SouthEast]  
![这里写图片描述][SouthEast 1]

### matlab爬虫实践 ###

fullURL = ['http://www.mathworks.com/matlabcentral/fileexchange' ...
           '?term=urlread'];
    str = urlread(fullURL,'Timeout',5);

fullURL = ['http://www.mathworks.com/matlabcentral/fileexchange' ...
           '?term=urlwrite'];
    filename = 'samples.html';
    urlwrite(fullURL,filename,'Timeout',5);

matlab的上述代码可以爬取网页源代码，接下来只需要对字符数据进行操作，即可得到图片数据地址，进行下载即可。

主要参考文献：  
[http://blog.csdn.net/baidu\_31956557/article/details/75579344][http_blog.csdn.net_baidu_31956557_article_details_75579344]  
[http://hongtoushizi.iteye.com/blog/2268825][http_hongtoushizi.iteye.com_blog_2268825]  
[http://blog.csdn.net/denny2015/article/details/50532968][http_blog.csdn.net_denny2015_article_details_50532968]  
[http://www.cnblogs.com/yinxiangnan-charles/p/5928689.html][http_www.cnblogs.com_yinxiangnan-charles_p_5928689.html]

[SouthEast]: /images/20220610/2fa2a4c1242a4a56b9c17ebb9ac1d185.png
[SouthEast 1]: /images/20220610/8b80a271dd0e415b981dce5bb3fff5c7.png
[http_blog.csdn.net_baidu_31956557_article_details_75579344]: http://blog.csdn.net/baidu_31956557/article/details/75579344
[http_hongtoushizi.iteye.com_blog_2268825]: http://hongtoushizi.iteye.com/blog/2268825
[http_blog.csdn.net_denny2015_article_details_50532968]: http://blog.csdn.net/denny2015/article/details/50532968
[http_www.cnblogs.com_yinxiangnan-charles_p_5928689.html]: http://www.cnblogs.com/yinxiangnan-charles/p/5928689.html