Jsoup爬取网络内容（包括图片文件），保存到本地和保存到数据库（一）

绝地灬酷狼 2022-12-20 15:54 363阅读 0赞

背景：

项目需要某个区县的天气数据，需要从中国气象局的官网中进行爬取。但是，中国气象局服务器调用接口返回的数据没有我想要的信息，比如说是未来24小时的天气温度，气压，风速等信息，这些数据接口中都没有，那么只能从页面中进行获取，所以使用了Jsoup，有一些信息是图片的形式返回，需要将图片下载下来存储到数据库中，用到的数据库是postgres数据库。工具类是Hutool和Jsoup

爬取的目标信息

target.xml

  1. <div id=day0 class="clearfix pull-left">
  2.     <div class="hour3 hbg">
  3.         <div> 11:00 </div>
  4.         <div class=hourimg>
  5.             <img src="http://image.nmc.cn/assets/img/w/40x40/3/0.png">
  6.             </div>
  7.             <div> - </div>
  8.             <div class=tmp_lte_20> 16.2℃ </div>
  9.             <div> 1.7m/s </div>
  10.             <div> 东北风 </div>
  11.             <div> 1002.1hPa </div>
  12.             <div> 56% </div>
  13.             <div class=hide> 0% </div>
  14.         </div>
  15.         <div class="hour3 hbg">
  16.             <div> 14:00 </div>
  17.             <div class=hourimg>
  18.                 <img src="http://image.nmc.cn/assets/img/w/40x40/3/0.png">
  19.                 </div>
  20.                 <div> - </div>
  21.                 <div class=tmp_lte_20> 17.8℃ </div>
  22.                 <div> 1.8m/s </div>
  23.                 <div> 东北风 </div>
  24.                 <div> 1000.5hPa </div>
  25.                 <div> 50.5% </div>
  26.                 <div class=hide> 0% </div>
  27.             </div>
  28.             <div class="hour3 hbg">
  29.                 <div> 17:00 </div>
  30.                 <div class=hourimg>
  31.                     <img src="http://image.nmc.cn/assets/img/w/40x40/3/0.png">
  32.                     </div>
  33.                     <div> - </div>
  34.                     <div class=tmp_lte_15> 13.1℃ </div>
  35.                     <div> 1.3m/s </div>
  36.                     <div> 东风 </div>
  37.                     <div> 1000.4hPa </div>
  38.                     <div> 61.6% </div>
  39.                     <div class=hide> 0% </div>
  40.                 </div>
  41.                 <div class="hour3 hbg">
  42.                     <div> 20:00 </div>
  43.                     <div class=hourimg>
  44.                         <img src="http://image.nmc.cn/assets/img/w/40x40/3/0.png">
  45.                         </div>
  46.                         <div> - </div>
  47.                         <div class=tmp_lte_10> 8.3℃ </div>
  48.                         <div> 0.8m/s </div>
  49.                         <div> 东风 </div>
  50.                         <div> 1002hPa </div>
  51.                         <div> 65.5% </div>
  52.                         <div class=hide> 0.2% </div>
  53.                     </div>
  54.                     <div class="hour3 hbg">
  55.                         <div> 23:00 </div>
  56.                         <div class=hourimg>
  57.                             <img src="http://image.nmc.cn/assets/img/w/40x40/3/0.png">
  58.                             </div>
  59.                             <div> - </div>
  60.                             <div class=tmp_lte_10> 6.6℃ </div>
  61.                             <div> 0.4m/s </div>
  62.                             <div> 东南风 </div>
  63.                             <div> 1001hPa </div>
  64.                             <div> 71.7% </div>
  65.                             <div class=hide> 10.1% </div>
  66.                         </div>
  67.                         <div class="hour3 ">
  68.                             <div> 14日02:00 </div>
  69.                             <div class=hourimg>
  70.                                 <img src="http://image.nmc.cn/assets/img/w/40x40/3/0.png">
  71.                                 </div>
  72.                                 <div> - </div>
  73.                                 <div class=tmp_lte_10> 6.2℃ </div>
  74.                                 <div> 0.5m/s </div>
  75.                                 <div> 南风 </div>
  76.                                 <div> 1000.5hPa </div>
  77.                                 <div> 81.9% </div>
  78.                                 <div class=hide> 10.9% </div>
  79.                             </div>
  80.                             <div class="hour3 ">
  81.                                 <div> 05:00 </div>
  82.                                 <div class=hourimg>
  83.                                     <img src="http://image.nmc.cn/assets/img/w/40x40/3/1.png">
  84.                                     </div>
  85.                                     <div> - </div>
  86.                                     <div class=tmp_lte_10> 7.8℃ </div>
  87.                                     <div> 0.7m/s </div>
  88.                                     <div> 南风 </div>
  89.                                     <div> 1000.1hPa </div>
  90.                                     <div> 93.6% </div>
  91.                                     <div class=hide> 52.6% </div>
  92.                                 </div>
  93.                                 <div class="hour3 ">
  94.                                     <div> 08:00 </div>
  95.                                     <div class=hourimg>
  96.                                         <img src="http://image.nmc.cn/assets/img/w/40x40/3/1.png">
  97.                                         </div>
  98.                                         <div> - </div>
  99.                                         <div class=tmp_lte_10> 9.1℃ </div>
  100.                                         <div> 0.9m/s </div>
  101.                                         <div> 南风 </div>
  102.                                         <div> 1000.3hPa </div>
  103.                                         <div> 84.5% </div>
  104.                                         <div class=hide> 61.4% </div>
  105.                                     </div>
  106.                                 </div>

爬取的目标数据展示

watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3BzaGRoeA_size_16_color_FFFFFF_t_70

建立实体类

TwentyFourHoursDomain.java

  1. package com.imegaware.crawler.weatherForecast.psd;
  3. import java.io.File;
  5. /**
  6.  * 24小时每隔三个小时需要的属性
  7.  * @author pshdhx
  8.  *
  9.  */
  10. public class TwentyFourHoursDomain {
  11.     private String monitorTime; //未来监测时间
  12.     private String url; //天气图标url ：http://image.nmc.cn/assets/img/w/40x40/3/0.png=晴天
  13.     private File file; //天气图标url ：http://image.nmc.cn/assets/img/w/40x40/3/0.png=晴天
  14.     private String rain; //降水 -
  15.     private String temperature; //气温
  16.     private String windSpeed; //风速
  17.     private String windDirection; //风向
  18.     private String airPressure; //气压
  19.     private String humidity;//湿度
  20.     private String url2; //本地url
  22.     public String getUrl2() {
  23.         return url2;
  24.     }
  25.     public void setUrl2(String url2) {
  26.         this.url2 = url2;
  27.     }
  28.     public String getMonitorTime() {
  29.         return monitorTime;
  30.     }
  31.     public void setMonitorTime(String monitorTime) {
  32.         this.monitorTime = monitorTime;
  33.     }
  34.     public String getUrl() {
  35.         return url;
  36.     }
  37.     public void setUrl(String url) {
  38.         this.url = url;
  39.     }
  41.     public File getFile() {
  42.         return file;
  43.     }
  44.     public void setFile(File file) {
  45.         this.file = file;
  46.     }
  47.     public String getRain() {
  48.         return rain;
  49.     }
  50.     public void setRain(String rain) {
  51.         this.rain = rain;
  52.     }
  53.     public String getTemperature() {
  54.         return temperature;
  55.     }
  56.     public void setTemperature(String temperature) {
  57.         this.temperature = temperature;
  58.     }
  59.     public String getWindSpeed() {
  60.         return windSpeed;
  61.     }
  62.     public void setWindSpeed(String windSpeed) {
  63.         this.windSpeed = windSpeed;
  64.     }
  65.     public String getWindDirection() {
  66.         return windDirection;
  67.     }
  68.     public void setWindDirection(String windDirection) {
  69.         this.windDirection = windDirection;
  70.     }
  71.     public String getAirPressure() {
  72.         return airPressure;
  73.     }
  74.     public void setAirPressure(String airPressure) {
  75.         this.airPressure = airPressure;
  76.     }
  77.     public String getHumidity() {
  78.         return humidity;
  79.     }
  80.     public void setHumidity(String humidity) {
  81.         this.humidity = humidity;
  82.     }
  83.     @Override
  84.     public String toString() {
  85.         return "TwentyFourHoursDomain [monitorTime=" + monitorTime + ", url=" + url + ", file=" + file + ", rain="
  86.                 + rain + ", temperature=" + temperature + ", windSpeed=" + windSpeed + ", windDirection="
  87.                 + windDirection + ", airPressure=" + airPressure + ", humidity=" + humidity + "]";
  88.     }
  90. }

JdbcUtil.java

  1. package com.imegaware.crawler.util;
  3. import java.io.InputStream;
  4. import java.sql.Connection;
  5. import java.sql.DriverManager;
  6. import java.sql.PreparedStatement;
  7. import java.sql.ResultSet;
  8. import java.sql.SQLException;
  9. import java.util.Properties;
  11. /**
  12.  * 
  13.  * @author pshdhx
  14.  *
  15.  */
  16. public class JdbcUtil {
  18.     private static String USERNAME;
  19.     private static String PASSWORD;
  20.     private static String DRIVER;
  21.     private static String URL;
  23.     static {
  24.         loadConfig();
  25.     }
  27.     /**
  28.      * 加载数据库配置信息
  29.      */
  30.     public static void loadConfig() {
  31.         try {
  32.             InputStream inStream = JdbcUtil.class.getClassLoader().getResourceAsStream("jdbc.properties");
  33.             Properties prop = new Properties();
  34.             prop.load(inStream);
  35.             USERNAME = prop.getProperty("jdbc.username");
  36.             PASSWORD = prop.getProperty("jdbc.password");
  37.             DRIVER = prop.getProperty("jdbc.driver");
  38.             URL = prop.getProperty("jdbc.url");
  39.         } catch (Exception e) {
  40.             throw new RuntimeException("读取数据库配置文件异常！");
  41.         }
  42.     }
  44.     public JdbcUtil() {
  45.         super();
  46.     }
  48.     /**
  49.      * 获取数据库连接
  50.      *
  51.      * @return
  52.      */
  53.     public static Connection getConnection() throws RuntimeException {
  54.         try {
  55.             Class.forName(DRIVER);
  56.             Connection connection = DriverManager.getConnection(URL, USERNAME, PASSWORD);
  57.             return connection;
  58.         } catch (Exception e) {
  59.             throw new RuntimeException("无法获取数据库连接！", e);
  60.         }
  61.     }
  63.     /**
  64.      * @Title close
  65.      * @Description 释放连接
  66.      * @author maven
  67.      * @param connection
  68.      * @param preparedStatement
  69.      * @param resultSet
  70.      * @return void
  71.      */
  72.     public static void close(Connection connection, PreparedStatement preparedStatement, ResultSet resultSet) {
  73.         if (resultSet != null) {
  74.             try {
  75.                 resultSet.close();
  76.             } catch (SQLException e) {
  77.                 e.printStackTrace();
  78.             }
  79.         }
  80.         if (preparedStatement != null) {
  81.             try {
  82.                 preparedStatement.close();
  83.             } catch (SQLException e) {
  84.                 e.printStackTrace();
  85.             }
  86.         }
  87.         if (connection != null) {
  88.             try {
  89.                 connection.close();
  90.             } catch (SQLException e) {
  91.                 e.printStackTrace();
  92.             }
  93.         }
  94.     }
  96.     public static void close(PreparedStatement preparedStatement, ResultSet resultSet) {
  97.         if (resultSet != null) {
  98.             try {
  99.                 resultSet.close();
  100.             } catch (SQLException e) {
  101.                 e.printStackTrace();
  102.             }
  103.         }
  105.         if (preparedStatement != null) {
  106.             try {
  107.                 preparedStatement.close();
  108.             } catch (SQLException e) {
  109.                 e.printStackTrace();
  110.             }
  111.         }
  112.     }
  114.     public static void close(PreparedStatement preparedStatement) {
  116.         if (preparedStatement != null) {
  117.             try {
  118.                 preparedStatement.close();
  119.             } catch (SQLException e) {
  120.                 e.printStackTrace();
  121.             }
  122.         }
  123.     }
  125.     public static void close(ResultSet resultSet) {
  126.         if (resultSet != null) {
  127.             try {
  128.                 resultSet.close();
  129.             } catch (SQLException e) {
  130.                 e.printStackTrace();
  131.             }
  132.         }
  133.     }
  135.     public static void close(Connection connection) {
  136.         if (connection != null) {
  137.             try {
  138.                 connection.close();
  139.             } catch (SQLException e) {
  140.                 e.printStackTrace();
  141.             }
  142.         }
  143.     }
  145. }

ImageUtil.java

  1. package com.imegaware.crawler.util;
  3. import java.io.File;
  4. import java.io.FileInputStream;
  5. import java.io.FileOutputStream;
  6. import java.io.IOException;
  7. import java.io.InputStream;
  8. /**
  9.  * 图片工具类
  10.  * @author pshdhx
  11.  *
  12.  */
  13. public class ImageUtil {
  14.     // 读取本地图片获取输入流
  15.     public static FileInputStream readImage(String path) throws IOException {
  16.         return new FileInputStream(new File(path));
  17.     }
  19.     // 读取表中图片获取输出流
  20.     public static void readBin2Image(InputStream in, String targetPath) {
  21.         File file = new File(targetPath);
  22.         String path = targetPath.substring(0, targetPath.lastIndexOf("/"));
  23.         if (!file.exists()) {
  24.             new File(path).mkdir();
  25.         }
  26.         FileOutputStream fos = null;
  27.         try {
  28.             fos = new FileOutputStream(file);
  29.             int len = 0;
  30.             byte[] buf = new byte[1024];
  31.             while ((len = in.read(buf)) != -1) {
  32.                 fos.write(buf, 0, len);
  33.             }
  34.            fos.flush();
  35.         } catch (Exception e) {
  36.             e.printStackTrace();
  37.         } finally {
  38.             if (null != fos) {
  39.                 try {
  40.                     fos.close();
  41.                 } catch (IOException e) {
  42.                     e.printStackTrace();
  43.                 }
  44.             }
  45.         }
  46.     }
  47. }

爬取主类

TwentyFourHoursWeather.java

  1. package com.imegaware.crawler.weatherForecast.psd;
  3. import java.io.File;
  4. import java.io.FileInputStream;
  5. import java.io.IOException;
  6. import java.sql.Connection;
  7. import java.sql.PreparedStatement;
  8. import java.sql.SQLException;
  9. import java.util.ArrayList;
  10. import java.util.List;
  12. import org.jsoup.Jsoup;
  13. import org.jsoup.nodes.Document;
  14. import org.jsoup.nodes.Element;
  15. import org.jsoup.nodes.Node;
  17. import com.alibaba.fastjson.JSONArray;
  18. import com.alibaba.fastjson.JSONObject;
  19. import com.imegaware.crawler.util.ImageUtil;
  20. import com.imegaware.crawler.util.JdbcUtil;
  21. import com.ruoyi.common.utils.StringUtils;
  23. import cn.hutool.core.io.FileUtil;
  24. import cn.hutool.http.HttpUtil;
  26. /**
  27.  * 利用jsoup爬取气象局未来24小时的天气情况，温度，湿度，降水，风速，风向，气压
  28.  * @author pshdhx
  29.  *
  30.  */
  31. public class TwentyFourHoursWeather {
  33.     //图片存储到本地的路径
  34.     private static String myurl = "/home/sdzw/JavaPaChong/ruoyi/imw-crawler/src/main/java/com/imegaware/crawler/weatherForecast/psd/";
  36.     private static List<TwentyFourHoursDomain> getItemContent() {
  37.         // TODO Auto-generated method stub
  38.         Document doc ;
  39.         List<TwentyFourHoursDomain> list = new ArrayList<>();
  40.         try {
  41.             doc = Jsoup.connect("http://www.nmc.cn/publish/forecast/ASD/zichuan.html").get();
  42.             Element day0 = doc.getElementById("day0");
  44.             for(int i=0;i<day0.childNodeSize();i++) {
  45.                 Node childNode = day0.childNode(i);
  46.                 //System.out.println(childNode.childNodeSize()+"--8个模块的子模块=需要的参数个数");
  47.                     TwentyFourHoursDomain pojo = new TwentyFourHoursDomain();
  48.                     pojo.setMonitorTime(StringUtils.trim(childNode.childNode(0).childNode(0)+""));
  49.                     String url = childNode.childNode(1).childNode(0).attr("src")+""; //网址
  50.                     long size = HttpUtil.downloadFile(url, FileUtil.file(myurl));
  51.                     //System.out.println("Download size: " + size);
  52.                     String[] split = url.split("[/]");
  53.                     File file = new File(myurl+split[split.length-1]);
  54.                     pojo.setFile(file);
  55.                     pojo.setUrl2(myurl+split[split.length-1]);
  56.                     //System.out.println(pojo.getUrl2()+"========"+"本地地址");
  57.                     pojo.setUrl(url);
  58.                     pojo.setRain(StringUtils.trim(childNode.childNode(2).childNode(0)+""));
  59.                     pojo.setTemperature(StringUtils.trim(childNode.childNode(3).childNode(0)+""));
  60.                     pojo.setWindSpeed(StringUtils.trim(childNode.childNode(4).childNode(0)+""));
  61.                     pojo.setWindDirection(StringUtils.trim(childNode.childNode(5).childNode(0)+""));
  62.                     pojo.setAirPressure(StringUtils.trim(childNode.childNode(6).childNode(0)+""));
  63.                     pojo.setHumidity(StringUtils.trim(childNode.childNode(7).childNode(0)+""));
  65.                 list.add(pojo);
  66.             }
  67.             saveData(list);
  69.         } catch (IOException e) {
  70.             // TODO Auto-generated catch block
  71.             e.printStackTrace();
  72.         }
  73.         return list;
  74.     }
  76.     /**
  77.      * @Title saveData
  78.      * @Description 数据入库
  79.      * @author maven
  80.      * @param param
  81.      * @return
  82.      * @return int
  83.      * @throws SQLException
  84.      */
  85.     private static void saveData(List<TwentyFourHoursDomain> list) throws RuntimeException {
  86.         int rows = 0;
  87.         PreparedStatement pstmt;
  88.         FileInputStream in = null;
  89.         try {
  90.             Connection conn = JdbcUtil.getConnection();
  91.             String sql = "INSERT INTO twenty_four_hour_weather(id,monitor_time,url,file,rain,temperature,wind_speed,wind_direction,air_pressure,humidity,region_name,region_code,task_time) VALUES (?,?,?,?,?,?,?,?,?,?,?,?,now());";
  92.             pstmt = conn.prepareStatement(sql);
  93.             for (int i = 0; i < list.size(); i++) {
  94.                 pstmt.setString(1, (i+100)*Math.random()*1000000+""+i+(i+10)*Math.random()*1000000);
  95.                 pstmt.setString(2, list.get(i).getMonitorTime()+"");
  96.                 pstmt.setString(3, list.get(i).getUrl());
  97.                 File file = list.get(i).getFile();
  98.                 in = ImageUtil.readImage(list.get(i).getUrl2());
  99.                 pstmt.setBinaryStream(4, in, in.available());
  100.                 pstmt.setString(5, list.get(i).getRain());
  101.                 pstmt.setString(6, list.get(i).getTemperature());
  102.                 pstmt.setString(7, list.get(i).getWindSpeed());
  103.                 pstmt.setString(8, list.get(i).getWindDirection());
  104.                 pstmt.setString(9, list.get(i).getAirPressure());
  105.                 pstmt.setString(10, list.get(i).getHumidity());
  106.                 pstmt.setString(11, "淄川区");
  107.                 pstmt.setString(12, "370302");
  108.                 pstmt.addBatch();
  109.             }
  110.             int[] x = pstmt.executeBatch();
  111.             JdbcUtil.close(conn, pstmt, null);
  112.             for (int i : x) {
  113.                 rows += i;
  114.             }
  115.             System.out.println("入库完成，共插入" + rows + "行数据");
  116.         } catch (Exception e) {
  117.             throw new RuntimeException("数据入库失败！", e);
  118.         }
  120.     }
  121.     public static void main(String[] args) {
  122.         TwentyFourHoursWeather test = new TwentyFourHoursWeather();
  123.         List<TwentyFourHoursDomain> itemContent = TwentyFourHoursWeather.getItemContent();
  124.         for(int i=0;i<itemContent.size();i++) {
  125.             System.out.println(itemContent.get(i).toString());
  126.         }
  127.         System.out.println(itemContent.size());
  128.     }
  129. }

pom.xml

  1. <dependency>
  2.             <groupId>org.jsoup</groupId>
  3.             <artifactId>jsoup</artifactId>
  4.             <version>1.11.3</version>
  5. </dependency>
  7. <dependency>
  8.             <groupId>cn.hutool</groupId>
  9.             <artifactId>hutool-all</artifactId>
  10.             <version>5.4.7</version>
  11. </dependency>

功能实现截图：

watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3BzaGRoeA_size_16_color_FFFFFF_t_70 1

发表评论

表情：
评论列表 （有 0 条评论，363人围观）

还没有评论，来说两句吧...

相关阅读