webmagic整理爬取例子

小咪咪 2022-07-14 05:25 201阅读 0赞

1.爬取csdn博客[http://blog.csdn.net/zhengyong15984285623/article/details/52865229][http_blog.csdn.net_zhengyong15984285623_article_details_52865229] 爬取网址为[http://blog.csdn.net/zhengyong15984285623][http_blog.csdn.net_zhengyong15984285623]

2、全注解爬取代理网站[![快代理logo][logo]][logo_logo][https://my.oschina.net/anxiaole/blog/755965][https_my.oschina.net_anxiaole_blog_755965]http://www.kuaidaili.com/free/

#### 三、 注意事项 ####

WebMagic使用log4j打印日志

<?xml version="1.0" encoding="UTF-8" ?>
    <!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">
    
    <log4j:configuration>
    
        <appender name="CONSOLE" class="org.apache.log4j.ConsoleAppender">
            <layout class="org.apache.log4j.PatternLayout">
                <param name="ConversionPattern"
                       value="%d - %c -%-4r [%t] %-5p %x - %m%n" />
            </layout>
    
            
            <filter class="org.apache.log4j.varia.LevelRangeFilter">
                <param name="LevelMax" value="ERROR"/>
                <param name="LevelMin" value="TRACE"/>
            </filter>
        </appender>
    
        <appender name="FILE" class="org.apache.log4j.FileAppender">
            <param name="File" value="/Users/zhengyong/log/crawl.log"/>
            <layout class="org.apache.log4j.PatternLayout">
                <param name="ConversionPattern"
                       value="%d - %c -%-4r [%t] %-5p %x - %m%n" />
            </layout>
        </appender>
    
        <root>
            <priority value="info" />
            <appender-ref ref="CONSOLE" />
            <appender-ref ref="FILE" />
        </root>
    
    
    </log4j:configuration>
     
      
       1 
       2 
       3 
       4 
       5 
       6 
       7 
       8 
       9 
       10 
       11 
       12 
       13 
       14 
       15 
       16 
       17 
       18 
       19 
       20 
       21 
       22 
       23 
       24 
       25 
       26 
       27 
       28 
       29 
       30 
       31 
       32 
       33 
       34 
      
     
      
       1 
       2 
       3 
       4 
       5 
       6 
       7 
       8 
       9 
       10 
       11 
       12 
       13 
       14 
       15 
       16 
       17 
       18 
       19 
       20 
       21 
       22 
       23 
       24 
       25 
       26 
       27 
       28 
       29 
       30 
       31 
       32 
       33 
       34

Java中main方法使用

import org.apache.log4j.xml.DOMConfigurator;
    
    DOMConfigurator.configure("/Users/zhengyong/crawl/log4j.xml");//加载.xml文件

[http_blog.csdn.net_zhengyong15984285623_article_details_52865229]: http://blog.csdn.net/zhengyong15984285623/article/details/52865229
[http_blog.csdn.net_zhengyong15984285623]: http://blog.csdn.net/zhengyong15984285623
[logo]: http://img.kuaidaili.com/img/kdl_logo.jpg?v=1
[logo_logo]: http://www.kuaidaili.com/
[https_my.oschina.net_anxiaole_blog_755965]: https://my.oschina.net/anxiaole/blog/755965