java爬虫Gecco爬取豆瓣top250的电影数据

心已赠人 2022-10-12 12:17 202阅读 0赞

### java爬虫Gecco爬取豆瓣top250的电影数据 ###

*   *  导入依赖
     *  获取所有url
     *  发起二次请求
     *  匹配二次请求的url，并存储数据
     *  处理二次请求的数据
     *  编写主函数，爬虫入口类
     *  参考

这几天，一个作业需要用到java爬虫，于是在这里记录一下

## 导入依赖 ##

<dependency>
                <groupId>com.geccocrawler</groupId>
                <artifactId>gecco</artifactId>
                <version>1.0.8</version>
            </dependency>

## 获取所有url ##

top250的url是那些链接  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzUyMDY3MA_size_16_color_FFFFFF_t_70]

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzUyMDY3MA_size_16_color_FFFFFF_t_70 1]  
我们可以写一个类来保存那些的url，如下parentName是保存选择器 “p” 文字信息，可要可不要；categorys是保存选择器 “a” 的url的；它的类型List <HrefBean，HrefBean如下图，它可以保存 a 标签的 href 和其包含的文字  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzUyMDY3MA_size_16_color_FFFFFF_t_70 2]  
Category类

package com.hzxy.reptile.crawl;
    
    import com.geccocrawler.gecco.annotation.HtmlField;
    import com.geccocrawler.gecco.annotation.Text;
    import com.geccocrawler.gecco.spider.HrefBean;
    import com.geccocrawler.gecco.spider.HtmlBean;
    
    import java.util.List;
    
    /** * 创建时间 2021/7/1 16:10 * 爬取数据的封装类 * @author Ye * @version 1.0 */
    public class Category implements HtmlBean { 
        private static final long serialVersionUID = 3018760488621382659L;
    
        @Text
        @HtmlField(cssPath="p")
        private String parentName;
    
        //保存url
        @HtmlField(cssPath="a")
        private List<HrefBean> categorys;
    
        public String getParentName() { 
            return parentName;
        }
    
        public void setParentName(String parentName) { 
            this.parentName = parentName;
        }
    
        public List<HrefBean> getCategorys() { 
            return categorys;
        }
    
        public void setCategorys(List<HrefBean> categorys) { 
            this.categorys = categorys;
        }
    
        @Override
        public String toString() { 
            return "Category{" +
                    "parentName='" + parentName + '\'' +
                    ", categorys=" + categorys +
                    '}';
        }
    }

这是获取所有url的类，匹配url https://movie.douban.com/top250，找到所有.info选择器的元素，把a的url赋值给Category的categorys，文字赋值给parentName  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzUyMDY3MA_size_16_color_FFFFFF_t_70 3]

AllSort类

## 发起二次请求 ##

如下图表示发起二次请求的代码，将allSort.getOne()的所有url都发送请求

package com.hzxy.reptile.crawl;
    
    import com.geccocrawler.gecco.annotation.PipelineName;
    import com.geccocrawler.gecco.pipeline.Pipeline;
    import com.geccocrawler.gecco.request.HttpRequest;
    import com.geccocrawler.gecco.scheduler.SchedulerContext;
    import com.geccocrawler.gecco.spider.HrefBean;
    
    import java.util.ArrayList;
    import java.util.List;
    
    /** * 创建时间 2021/7/1 16:39 * 获取到启动类的url的处理类 * @author Ye * @version 1.0 */
    @PipelineName("allSortPipeline")
    public class AllSortPipeline implements Pipeline<AllSort> { 
    
        @Override
        public void process(AllSort allSort) { 
                System.out.println("-=======-");
                List<Category> categorys = new ArrayList<>();
                categorys.addAll(allSort.getOne());
                for(Category category : categorys) { 
                    System.out.println(category);
                    for (HrefBean hrefBean:category.getCategorys()){ 
                        System.out.println(hrefBean.getTitle()+" "+hrefBean.getUrl());
                        String url = hrefBean.getUrl();
                        //发送二级请求
                        HttpRequest currRequest = allSort.getRequest();
                        SchedulerContext.into(currRequest.subRequest(url));
                    }
                }
        }
    }

输出信息  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzUyMDY3MA_size_16_color_FFFFFF_t_70 4]

## 匹配二次请求的url，并存储数据 ##

根据第一步爬的数据，我们匹配https://movie.douban.com/subject/\{url\}/，只要是符合的，我们就会发送请求，解析页面将信息保存在这个实体。

解析小技巧，点击那个箭头，拖到要解析元素里  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzUyMDY3MA_size_16_color_FFFFFF_t_70 5]  
然后右键，copy，再copy selector，即可找到选择器  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzUyMDY3MA_size_16_color_FFFFFF_t_70 6]

具体代码如下：

package com.hzxy.reptile.crawl;
    
    import com.geccocrawler.gecco.annotation.Gecco;
    import com.geccocrawler.gecco.annotation.HtmlField;
    import com.geccocrawler.gecco.annotation.RequestParameter;
    import com.geccocrawler.gecco.annotation.Text;
    import com.geccocrawler.gecco.spider.HtmlBean;
    
    /** * 创建时间 2021/7/1 23:42 * 获取详细数据的实体类 * @author Ye * @version 1.0 */
    
    //匹配二次请求的url
    @Gecco(matchUrl="https://movie.douban.com/subject/{url}/", pipelines={ "consolePipeline", "movieDetailPipeline"})
    public class MovieDetail implements HtmlBean { 
    
        private static final long serialVersionUID = -377053120283382723L;
        
        //url里面的url
        @RequestParameter
        private String url;
    
        //电影名
        @Text
        @HtmlField(cssPath="#content > h1 > span:nth-child(1)")
        private String title;
    
        //导演
        @Text
        @HtmlField(cssPath="#info > span:nth-child(1) > span.attrs > a")
        private String director;
    
        //编剧
        @Text
        @HtmlField(cssPath="#info > span:nth-child(3) > span.attrs > a:nth-child(1)")
        private String writer;
    
        //类型
        @Text
        @HtmlField(cssPath="#info > span:nth-child(8)")
        private String type;
    
        //类型
        @Text
        @HtmlField(cssPath="#info > span:nth-child(9)")
        private String type1;
    
        //评价人数
        @Text
        @HtmlField(cssPath="#interest_sectl > div.rating_wrap.clearbox > div.rating_self.clearfix > div > div.rating_sum > a > span")
        private String number;
    
        //评价人数
        @Text
        @HtmlField(cssPath="#interest_sectl > div.rating_wrap.clearbox > div.rating_self.clearfix > strong")
        private String score;
    
    
        public String getUrl() { 
            return url;
        }
    
        public void setUrl(String url) { 
            this.url = url;
        }
    
        public String getTitle() { 
            return title;
        }
    
        public void setTitle(String title) { 
            this.title = title;
        }
    
        public String getDirector() { 
            return director;
        }
    
        public void setDirector(String director) { 
            this.director = director;
        }
    
        public String getWriter() { 
            return writer;
        }
    
        public void setWriter(String writer) { 
            this.writer = writer;
        }
    
        public String getType() { 
            return type;
        }
    
        public void setType(String type) { 
            this.type = type;
        }
    
        public String getType1() { 
            return type1;
        }
    
        public void setType1(String type1) { 
            this.type1 = type1;
        }
    
        public String getNumber() { 
            return number;
        }
    
        public void setNumber(String number) { 
            this.number = number;
        }
    
        public String getScore() { 
            return score;
        }
    
        public void setScore(String score) { 
            this.score = score;
        }
    
        @Override
        public String toString() { 
            return "ProductDetail{" +
                    "url='" + url + '\'' +
                    ", title='" + title + '\'' +
                    ", director='" + director + '\'' +
                    ", writer='" + writer + '\'' +
                    ", type='" + type + '\'' +
                    ", type1='" + type1 + '\'' +
                    ", number='" + number + '\'' +
                    ", score='" + score + '\'' +
                    '}';
        }
    }

## 处理二次请求的数据 ##

在这里，我们将二次请求解析到的数据输出

package com.hzxy.reptile.crawl;
    
    import com.geccocrawler.gecco.annotation.PipelineName;
    import com.geccocrawler.gecco.pipeline.Pipeline;
    
    /** * 创建时间 2021/7/2 0:07 * 对爬取到的数据的操作类 * @author Ye * @version 1.0 */
    //对二次请求爬取的url进行处理
    @PipelineName("movieDetailPipeline")
    public class MovieDetailPipeline implements Pipeline<MovieDetail> { 
    
        @Override
        public void process(MovieDetail movieDetail) { 
            System.out.println("~~~~~~~~~movieDetailPipeline~~~~~~~~~~~");
            System.out.println(movieDetail);
        }
    }

## 编写主函数，爬虫入口类 ##

package com.hzxy.reptile.crawl;
    
    import com.geccocrawler.gecco.GeccoEngine;
    import com.geccocrawler.gecco.request.HttpGetRequest;
    
    /** * 创建时间 2021/7/1 16:47 * 爬虫入口类 * @author Ye * @version 1.0 */
    public class Main { 
    
        public static void main(String[] rags) { 
            System.out.println("=======start========");
            HttpGetRequest startUrl = new HttpGetRequest("https://movie.douban.com/top250");
            startUrl.setCharset("GBK");
            GeccoEngine.create()
                    //Gecco搜索的包路径
                    .classpath("com.hzxy.reptile.crawl")
                    //开始抓取的页面地址
                    .start(startUrl)
                    //开启几个爬虫线程
                    .thread(1)
                    //单个爬虫每次抓取完一个请求后的间隔时间
                    .interval(1000)
                    .run();
        }
    }

结果如图：  
![在这里插入图片描述][202107042212455.png]  
![在这里插入图片描述][20210704221310103.png]  
最后爬到了我们想要的数据！

## 参考 ##

http://www.geccocrawler.com/

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzUyMDY3MA_size_16_color_FFFFFF_t_70]: /images/20221005/54684490f99449c3a7c50ac36031396d.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzUyMDY3MA_size_16_color_FFFFFF_t_70 1]: /images/20221005/9a0c01f1a43147d1981896e99b29d1fa.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzUyMDY3MA_size_16_color_FFFFFF_t_70 2]: /images/20221005/58ddaeea6ac744db95fc04617dbf3d67.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzUyMDY3MA_size_16_color_FFFFFF_t_70 3]: /images/20221005/3b9ec6dd5abb4d8093d7907061ef4272.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzUyMDY3MA_size_16_color_FFFFFF_t_70 4]: /images/20221005/0b25eeeac15d4d5a8ca618ef89ac8ef5.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzUyMDY3MA_size_16_color_FFFFFF_t_70 5]: /images/20221005/3aee313ca33d40918f321e4247b68394.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzUyMDY3MA_size_16_color_FFFFFF_t_70 6]: /images/20221005/30f02415905642cdabf1407d7fdf54a4.png
[202107042212455.png]: /images/20221005/b01c90d059b34283a55e8ba92748beb3.png
[20210704221310103.png]: /images/20221005/1fd35ce97acd4de393bb75f101f1a7e8.png