博客搬家系列（三）-爬取博客园博客

╰半夏微凉° 2022-04-22 01:32 346阅读 0赞

博客搬家系列（三）-爬取博客园博客

## 一.前情回顾 ##

博客搬家系列（一）-简介：[https://blog.csdn.net/rico\_zhou/article/details/83619152][https_blog.csdn.net_rico_zhou_article_details_83619152]

博客搬家系列（二）-爬取CSDN博客：[https://blog.csdn.net/rico\_zhou/article/details/83619509][https_blog.csdn.net_rico_zhou_article_details_83619509]

博客搬家系列（四）-爬取简书文章：[https://blog.csdn.net/rico\_zhou/article/details/83619538][https_blog.csdn.net_rico_zhou_article_details_83619538]

博客搬家系列（五）-爬取开源中国博客：[https://blog.csdn.net/rico\_zhou/article/details/83619561][https_blog.csdn.net_rico_zhou_article_details_83619561]

博客搬家系列（六）-爬取今日头条文章：[https://blog.csdn.net/rico\_zhou/article/details/83619564][https_blog.csdn.net_rico_zhou_article_details_83619564]

博客搬家系列（七）-本地WORD文档转HTML：[https://blog.csdn.net/rico\_zhou/article/details/83619573][https_blog.csdn.net_rico_zhou_article_details_83619573]

博客搬家系列（八）-总结：[https://blog.csdn.net/rico\_zhou/article/details/83619599][https_blog.csdn.net_rico_zhou_article_details_83619599]

## 二.开干（获取文章URL集合） ##

爬取博客园的博客思路跟CSDN一样，且下载图片那一步更为简单，任何header都不需要设置，同样，我们以ricozhou的主页为例分析源码[https://www.cnblogs.com/ricozhou/][https_www.cnblogs.com_ricozhou]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3JpY29femhvdQ_size_16_color_FFFFFF_t_70][]

我们可以看到文章列表如下，依然是很简洁的url，我们找一个博主文章较多的看看，方便分析规律，如[https://www.cnblogs.com/xdp-gacl/][https_www.cnblogs.com_xdp-gacl]

当我们点击下一页的时候，url如下

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3JpY29femhvdQ_size_16_color_FFFFFF_t_70 1][]

显然最后的2是页数，这样我们就找到了页面url规律，同样右击查看源码，分析找到文章都位于哪个标签内

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3JpY29femhvdQ_size_16_color_FFFFFF_t_70 2][]

观察发现，文章url均位于class为postTitle的标签内，代码如下：

/**
    	 * @date Oct 17, 2018 12:30:46 PM
    	 * @Desc
    	 * @param blogMove
    	 * @param oneUrl
    	 * @return
    	 * @throws IOException
    	 * @throws MalformedURLException
    	 * @throws FailingHttpStatusCodeException
    	 */
    	public void getCnBlogArticleUrlList(Blogmove blogMove, String oneUrl, List<String> urlList)
    			throws FailingHttpStatusCodeException, MalformedURLException, IOException {
    		// 模拟浏览器操作
    		// 创建WebClient
    		WebClient webClient = new WebClient(BrowserVersion.CHROME);
    		// 关闭css代码功能
    		webClient.getOptions().setThrowExceptionOnScriptError(false);
    		webClient.getOptions().setCssEnabled(false);
    		// 如若有可能找不到文件js则加上这句代码
    		webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
    		// 获取第一级网页html
    		HtmlPage page = webClient.getPage(oneUrl);
    		// System.out.println(page.asXml());
    		Document doc = Jsoup.parse(page.asXml());
    		Elements pageMsg = doc.select("div.postTitle");
    		Element linkNode;
    		for (Element e : pageMsg) {
    			linkNode = e.select("a.postTitle2").first();
    			if (linkNode == null) {
    				continue;
    			}
    			if (urlList.size() < blogMove.getMoveNum()) {
    				urlList.add(linkNode.attr("href"));
    			} else {
    				break;
    			}
    		}
    		return;
    	}

获取url集合如下

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3JpY29femhvdQ_size_16_color_FFFFFF_t_70 3][]

## 三.开干（获取文章具体信息） ##

同样，我们还是打开一篇博文，以[使用爬虫框架htmlunit整合springboot出现的一个不兼容问题][htmlunit_springboot] 为例，使用Chrome打开，我们可以看到一些基本信息，如文章的类型为原创，标题，时间，作者，阅读数，文章文字信息，图片信息等

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3JpY29femhvdQ_size_16_color_FFFFFF_t_70 4][]

![2018110115560986.png][]

同样，右击查看源码找到对应的元素，然后获取内容

![20181101155829105.png][]

![2018110115583396.png][]

部分代码

/**
    	 * @date Oct 17, 2018 12:46:52 PM
    	 * @Desc 获取详细信息
    	 * @param blogMove
    	 * @param url
    	 * @return
    	 * @throws IOException
    	 * @throws MalformedURLException
    	 * @throws FailingHttpStatusCodeException
    	 */
    	public Blogcontent getCnBlogArticleMsg(Blogmove blogMove, String url, List<Blogcontent> bList)
    			throws FailingHttpStatusCodeException, MalformedURLException, IOException {
    		Blogcontent blogcontent = new Blogcontent();
    		blogcontent.setArticleSource(blogMove.getMoveWebsiteId());
    		// 模拟浏览器操作
    		// 创建WebClient
    		WebClient webClient = new WebClient(BrowserVersion.CHROME);
    		// 关闭css代码功能
    		webClient.getOptions().setThrowExceptionOnScriptError(false);
    		webClient.getOptions().setCssEnabled(false);
    		// 如若有可能找不到文件js则加上这句代码
    		webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
    		// 获取第一级网页html
    		HtmlPage page = webClient.getPage(url);
    
    		Document doc = Jsoup.parse(page.asXml());
    		// 获取标题
    		String title = BlogMoveCnBlogUtils.getCnBlogArticleTitle(doc);
    		// 是否重复去掉
    		if (blogMove.getMoveRemoveRepeat() == 0) {
    			// 判断是否重复
    			if (BlogMoveCommonUtils.articleRepeat(bList, title)) {
    				return null;
    			}
    		}
    		blogcontent.setTitle(title);
    		// 获取作者
    		blogcontent.setAuthor(BlogMoveCnBlogUtils.getCnBlogArticleAuthor(doc));
    		// 获取时间
    		if (blogMove.getMoveUseOriginalTime() == 0) {
    			blogcontent.setGtmCreate(BlogMoveCnBlogUtils.getCnBlogArticleTime(doc));
    		} else {
    			blogcontent.setGtmCreate(new Date());
    		}
    		blogcontent.setGtmModified(new Date());
    		// 获取类型
    		blogcontent.setType(BlogMoveCnBlogUtils.getCnBlogArticleType(doc));
    		// 获取正文
    		blogcontent.setContent(BlogMoveCnBlogUtils.getCnBlogArticleContent(doc, blogMove, blogcontent));
    
    		// 设置其他
    		blogcontent.setStatus(blogMove.getMoveBlogStatus());
    		blogcontent.setBlogColumnName(blogMove.getMoveColumn());
    		// 特殊处理
    		blogcontent.setArticleEditor(blogMove.getMoveArticleEditor());
    		blogcontent.setShowId(DateUtils.format(new Date(), DateUtils.YYYYMMDDHHMMSSSSS));
    		blogcontent.setAllowComment(0);
    		blogcontent.setAllowPing(0);
    		blogcontent.setAllowDownload(0);
    		blogcontent.setShowIntroduction(1);
    		blogcontent.setIntroduction("");
    		blogcontent.setPrivateArticle(1);
    
    		return blogcontent;
    	}

详细信息

/**
    	 * @date Oct 17, 2018 1:10:19 PM
    	 * @Desc 获取标题
    	 * @param doc
    	 * @return
    	 */
    	public static String getCnBlogArticleTitle(Document doc) {
    		// 标题
    		Element pageMsg2 = doc.select("div#post_detail").first().select("h1.postTitle").first().select("a").first();
    		return pageMsg2.ownText();
    	}
    
    	/**
    	 * @date Oct 17, 2018 1:10:28 PM
    	 * @Desc 获取作者
    	 * @param doc
    	 * @return
    	 */
    	public static String getCnBlogArticleAuthor(Document doc) {
    		Element pageMsg2 = doc.select("div.postDesc").first().select("a").first();
    		return pageMsg2.ownText();
    	}
    
    	/**
    	 * @date Oct 17, 2018 1:10:33 PM
    	 * @Desc 获取时间
    	 * @param doc
    	 * @return
    	 */
    	public static Date getCnBlogArticleTime(Document doc) {
    		Element pageMsg2 = doc.select("div.postDesc").first().select("span#post-date").first();
    		String date = pageMsg2.ownText().trim();
    		// 这地方时间格式变化太多暂时不实现
    		Date d = DateUtils.formatStringDate(date, DateUtils.YYYY_MM_DD_HH_MM_SS4);
    		// 注意有些格式不正确
    		return d == null ? new Date() : d;
    	}
    
    	/**
    	 * @date Oct 17, 2018 1:10:37 PM
    	 * @Desc 获取类型
    	 * @param doc
    	 * @return
    	 */
    	public static String getCnBlogArticleType(Document doc) {
    		// Element pageMsg2 =
    		// doc.select("div.article-detail").first().select("h1.header").first().select("div.horizontal")
    		// .first();
    		// if ("原".equals(pageMsg2.html())) {
    		// return "原创";
    		// } else if ("转".equals(pageMsg2.html())) {
    		// return "转载";
    		// } else if ("译".equals(pageMsg2.html())) {
    		// return "翻译";
    		// }
    		return "原创";
    	}

/**
    	 * @date Oct 17, 2018 1:10:41 PM
    	 * @Desc 获取正文
    	 * @param doc
    	 * @param object
    	 * @param blogcontent
    	 * @return
    	 */
    	public static String getCnBlogArticleContent(Document doc, Blogmove blogMove, Blogcontent blogcontent) {
    		Element pageMsg2 = doc.select("div#post_detail").first().select("div#cnblogs_post_body").first();
    		String content = pageMsg2.toString();
    		String images;
    		// 注意是否需要替换图片
    		if (blogMove.getMoveSaveImg() == 0) {
    			// 保存图片到本地
    			// 先获取所有图片连接，再按照每个链接下载图片，最后替换原有链接
    			// 先创建一个文件夹
    			// 先创建一个临时文件夹
    			String blogFileName = String.valueOf(UUID.randomUUID());
    			FileUtils.createFolder(FilePathConfig.getUploadBlogPath() + File.separator + blogFileName);
    			blogcontent.setBlogFileName(blogFileName);
    			// 匹配出所有链接
    			List<String> imgList = BlogMoveCommonUtils.getArticleImgList(content);
    			// 下载并返回重新生成的imgurllist
    			List<String> newImgList = BlogMoveCommonUtils.getArticleNewImgList(blogMove, imgList, blogFileName);
    			// 拼接文章所有链接
    			images = BlogMoveCommonUtils.getArticleImages(newImgList);
    			blogcontent.setImages(images);
    			// 替换所有链接按顺序
    			content = getCnBlogNewArticleContent(content, imgList, newImgList);
    
    		}
    
    		return content;
    	}

代码共用，不再多放。还是一样的步骤，获取正文源码html匹配img，下载img，替换img链接，返回替换后的HTML

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3JpY29femhvdQ_size_16_color_FFFFFF_t_70 5][]

本人网站效果图：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3JpY29femhvdQ_size_16_color_FFFFFF_t_70 6][]

欢迎交流学习！

完整源码请见github：[https://github.com/ricozhou/blogmove][https_github.com_ricozhou_blogmove]

[https_blog.csdn.net_rico_zhou_article_details_83619152]: https://blog.csdn.net/rico_zhou/article/details/83619152
[https_blog.csdn.net_rico_zhou_article_details_83619509]: https://blog.csdn.net/rico_zhou/article/details/83619509
[https_blog.csdn.net_rico_zhou_article_details_83619538]: https://blog.csdn.net/rico_zhou/article/details/83619538
[https_blog.csdn.net_rico_zhou_article_details_83619561]: https://blog.csdn.net/rico_zhou/article/details/83619561
[https_blog.csdn.net_rico_zhou_article_details_83619564]: https://blog.csdn.net/rico_zhou/article/details/83619564
[https_blog.csdn.net_rico_zhou_article_details_83619573]: https://blog.csdn.net/rico_zhou/article/details/83619573
[https_blog.csdn.net_rico_zhou_article_details_83619599]: https://blog.csdn.net/rico_zhou/article/details/83619599
[https_www.cnblogs.com_ricozhou]: https://www.cnblogs.com/ricozhou/
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3JpY29femhvdQ_size_16_color_FFFFFF_t_70]: /images/20220422/4a46201f8bc44063a60ba17edc24eae5.png
[https_www.cnblogs.com_xdp-gacl]: https://www.cnblogs.com/xdp-gacl/
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3JpY29femhvdQ_size_16_color_FFFFFF_t_70 1]: /images/20220422/c718504f62294d39960a24de5151e85b.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3JpY29femhvdQ_size_16_color_FFFFFF_t_70 2]: /images/20220422/6eaab2881b854e64a27f56ef68cb7a81.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3JpY29femhvdQ_size_16_color_FFFFFF_t_70 3]: /images/20220422/5f15adfc5d624c61b258a63ba53c0b57.png
[htmlunit_springboot]: https://www.cnblogs.com/ricozhou/p/9878014.html
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3JpY29femhvdQ_size_16_color_FFFFFF_t_70 4]: /images/20220422/f4915abe3e48427fb50c04e0a2af7886.png
[2018110115560986.png]: /images/20220422/450ae99054f84df197a38e33a42fc098.png
[20181101155829105.png]: /images/20220422/6623093f2f9146e09f26fd8fc4453119.png
[2018110115583396.png]: /images/20220422/30614d6533b146f1abdfb4ac9039bb57.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3JpY29femhvdQ_size_16_color_FFFFFF_t_70 5]: /images/20220422/b6ad9dc318824b878c125c97b3ccf41c.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3JpY29femhvdQ_size_16_color_FFFFFF_t_70 6]: /images/20220422/d60da6dfebd84358a8ef512d29db9828.png
[https_github.com_ricozhou_blogmove]: https://github.com/ricozhou/blogmove