java爬虫好的教程_[Java教程]一个更加简单粗暴的爬虫

古城微笑少年丶 2022-11-10 05:43 133阅读 0赞

\[Java教程\]一个更加简单粗暴的爬虫

2017-09-30 16:00:13

今天上午看到一篇文章：一个简单粗暴的爬虫 - 必应今日美图。我也用自己的方式更加简单的实现了这个功能。下面我就贴一下自己的代码和思路。

我就不分析原博的思路了，原博写的很清楚。我用的是htmlunit，从原博的第三步开始，观察三级地址的规律， > 1 import com.gargoylesoftware.htmlunit.FailingHttpStatusCodeException; 2 import com.gargoylesoftware.htmlunit.WebClient; 3 import com.gargoylesoftware.htmlunit.html.HtmlAnchor; 4 import com.gargoylesoftware.htmlunit.html.HtmlPage; 5 import org.apache.commons.io.FileUtils; 6 7 import java.io.File; 8 import java.io.IOException; 9 import java.io.InputStream;10 import java.net.MalformedURLException;11 12 /\*\*13 \*

14  * class intro15  *

16 \*

17 \* Created by fujun on 2017/9/29.18 \*/19 public class BingPhoto \{20 public static void main(String\[\] args) \{21 WebClient webClient = new WebClient();22 webClient.getOptions().setCssEnabled(false);23 webClient.getOptions().setJavaScriptEnabled(false);24 for (int i = 1; i < 491; i++) \{25 String url = "http://bing.plmeizi.com/show/" + i;26 try \{27 HtmlPage page = webClient.getPage(url);28 HtmlAnchor a = (HtmlAnchor) page.getElementById("picurl");29 String picUrl = a.getHrefAttribute();30 if (picUrl.contains("jpg")) \{31 InputStream inputStream = webClient.getPage(picUrl).getWebResponse().getContentAsStream();32 FileUtils.copyToFile(inputStream, new File("E://bingpic//" + i + ".jpg"));33 System.out.println("SUCCEED:" + url);34 \} else \{35 System.out.println("FAILED:" + url);36 \}37 38 \} catch (FailingHttpStatusCodeException e) \{39 // TODO Auto-generated catch block40 e.printStackTrace();41 \} catch (MalformedURLException e) \{42 // TODO Auto-generated catch block43 e.printStackTrace();44 \} catch (IOException e) \{45 // TODO Auto-generated catch block46 e.printStackTrace();47 \} finally \{48 webClient.close(); // 关闭客户端，释放内存49 \}50 \}51 \}52 \}

注意点：

有些页面是404的，例如 src='/images/loading.gif' data-original="http://images2017.cnblogs.com/blog/1250684/201709/1250684-20170930150215075-698272723.png" alt="">

本文网址：http://www.shaoqun.com/a/327023.html

\*特别声明：以上内容来自于网络收集，著作权属原作者所有，如有侵权，请联系我们：admin@shaoqun.com。