爬虫如何实现每天爬取，定点爬取[以股票数据为例]

矫情吗；* 2022-07-11 08:40 756阅读 0赞
  • 分析抓取的数据
  • 抓包
  • 框架
  • model
  • main
  • util
  • parse
  • db
  • 问题所在

  • 解决方法

    • job
    • jobmain

近期，有人将本人博客，复制下来，直接上传到百度文库等平台。
本文为原创博客，仅供技术学习使用。未经允许，禁止将其复制下来上传到百度文库等平台。如有转载请注明本文博客的地址(链接)

分析抓取的数据

本文是以东方财富网的数据为例，这里只做技术学习使用，请勿滥用。如本文要抓取的数据是东方财富网的汽车板块及石油板块数据。如下为其地址：http://quote.eastmoney.com/center/list.html#28002481_0_2
http://quote.eastmoney.com/center/list.html#28002464_0_2
如下截图为其数据格式。

这里写图片描述

抓包

写爬虫第一步是做网络抓包，这个我之前的博客中已经讲到。即看数据请求的真实地址。关于本文为什么这样设计，请看我的专题博客，爬虫原理及相关基础：http://blog.csdn.net/column/details/14269.html。
这里写图片描述

从上图中，可以看出数据真实的请求地址及请求的方法。而获得的是json数组。如下图所示：
这里写图片描述

框架

本文使用的框架，如下图所示：
这里写图片描述

db:主要放的是数据库操作文件，包含MyDataSource【数据库驱动注册、连接数据库的用户名、密码】，MYSQLControl【连接数据库，插入操作、更新操作、建表操作等】。

model：用来封装对象，说的直白一些，封装的就是我要操作数据对应的属性名。有不明白的看之前写的一个简单的网络爬虫(http://blog.csdn.net/qy20115549/article/details/52203722)。

parse：这里面存放的是针对util获取的文件，进行解析，一般采用Jsoup解析；若是针对json数据，可采用正则表达式或者fastjson工具进行解析，建议使用fastjson，因其操作简单，快捷。

main：程序起点，也是重点，获取数据，执行数据库语句，存放数据。

job：用来执行的job任务。

jobmain：控制器，即合适执行job，如本文中的每天执行一次job。股票数据每天下午3点钟收盘，即设置为3点钟以后的某个时间点开始爬行相关股票数据。

model

model用来封装我要爬去的数据，如当天的日期，股票的id，股票的名称，股票价格等等。如下面程序：

  1. package model;
  2. /** * @author:合肥工业大学 管理学院 钱洋 * @email:1563178220@qq.com * @ */
  3. public class ExtMarketOilStockModel { 
  4.     private String date;
  5.     private String stock_id;
  6.     private String stock_name;
  7.     private float stock_price;
  8.     private float stock_change;
  9.     private float stock_range;
  10.     private float stock_amplitude;
  11.     private int stock_trading_number;
  12.     private int stock_trading_value;
  13.     private float stock_yesterdayfinish_price;
  14.     private float stock_todaystart_price;
  15.     private float stock_max_price;
  16.     private float stock_min_price;
  17.     private float stock_fiveminuate_change;
  18.     private String craw_time;
  19.     public String getDate() {
  20.         return date;
  21.     }
  22.     public void setDate(String date) {
  23.         this.date = date;
  24.     }
  26.     public String getStock_id() {
  27.         return stock_id;
  28.     }
  29.     public void setStock_id(String stock_id) {
  30.         this.stock_id = stock_id;
  31.     }
  32.     public String getStock_name() {
  33.         return stock_name;
  34.     }
  35.     public void setStock_name(String stock_name) {
  36.         this.stock_name = stock_name;
  37.     }
  38.     public float getStock_price() {
  39.         return stock_price;
  40.     }
  41.     public void setStock_price(float stock_price) {
  42.         this.stock_price = stock_price;
  43.     }
  44.     public float getStock_change() {
  45.         return stock_change;
  46.     }
  47.     public void setStock_change(float stock_change) {
  48.         this.stock_change = stock_change;
  49.     }
  50.     public float getStock_range() {
  51.         return stock_range;
  52.     }
  53.     public void setStock_range(float stock_range) {
  54.         this.stock_range = stock_range;
  55.     }
  56.     public float getStock_amplitude() {
  57.         return stock_amplitude;
  58.     }
  59.     public void setStock_amplitude(float stock_amplitude) {
  60.         this.stock_amplitude = stock_amplitude;
  61.     }
  63.     public int getStock_trading_number() {
  64.         return stock_trading_number;
  65.     }
  66.     public void setStock_trading_number(int stock_trading_number) {
  67.         this.stock_trading_number = stock_trading_number;
  68.     }
  69.     public int getStock_trading_value() {
  70.         return stock_trading_value;
  71.     }
  72.     public void setStock_trading_value(int stock_trading_value) {
  73.         this.stock_trading_value = stock_trading_value;
  74.     }
  75.     public float getStock_yesterdayfinish_price() {
  76.         return stock_yesterdayfinish_price;
  77.     }
  78.     public void setStock_yesterdayfinish_price(float stock_yesterdayfinish_price) {
  79.         this.stock_yesterdayfinish_price = stock_yesterdayfinish_price;
  80.     }
  81.     public float getStock_todaystart_price() {
  82.         return stock_todaystart_price;
  83.     }
  84.     public void setStock_todaystart_price(float stock_todaystart_price) {
  85.         this.stock_todaystart_price = stock_todaystart_price;
  86.     }
  87.     public float getStock_max_price() {
  88.         return stock_max_price;
  89.     }
  90.     public void setStock_max_price(float stock_max_price) {
  91.         this.stock_max_price = stock_max_price;
  92.     }
  93.     public float getStock_min_price() {
  94.         return stock_min_price;
  95.     }
  96.     public void setStock_min_price(float stock_min_price) {
  97.         this.stock_min_price = stock_min_price;
  98.     }
  99.     public float getStock_fiveminuate_change() {
  100.         return stock_fiveminuate_change;
  101.     }
  102.     public void setStock_fiveminuate_change(float stock_fiveminuate_change) {
  103.         this.stock_fiveminuate_change = stock_fiveminuate_change;
  104.     }
  105.     public String getCraw_time() {
  106.         return craw_time;
  107.     }
  108.     public void setCraw_time(String craw_time) {
  109.         this.craw_time = craw_time;
  110.     }
  111. }

main

主方法，尽量要求简单，这里我就这样写了。这里面有注释，很好理解。

  1. package navi.main;
  2. /** * @author:合肥工业大学 管理学院 钱洋 * @email:1563178220@qq.com * @ */
  3. import java.util.ArrayList;
  4. import java.util.List;
  6. import db.MYSQLControl;
  7. import model.ExtMarketOilStockModel;
  8. import parse.ExtMarketOilStockParse;
  10. public class ExtMarketOilStockMain { 
  12.     public static void main(String[] args) throws Exception {
  13.         List<String> urloillist=new ArrayList<String>();
  14.         List<String> urlcarlist=new ArrayList<String>();
  15.         List<ExtMarketOilStockModel> oilstocks=new ArrayList<ExtMarketOilStockModel>();
  16.         List<ExtMarketOilStockModel> carstocks=new ArrayList<ExtMarketOilStockModel>();
  17.         //石油相关股票就两页，对应两个地址
  18.         String url1="http://nufm.dfcfw.com/EM_Finance2014NumericApplication/JS.aspx?type=CT&cmd=C.BK04641&sty=FCOIATA&sortType=C&sortRule=-1&page=1&pageSize=20&js=var%20quote_123%3d{rank:[(x)],pages:(pc)}&token=7bc05d0d4c3c22ef9fca8c2a912d779c&jsName=quote_123&_g=0.13204790262127375";
  19.         String url2="http://nufm.dfcfw.com/EM_Finance2014NumericApplication/JS.aspx?type=CT&cmd=C.BK04641&sty=FCOIATA&sortType=C&sortRule=-1&page=2&pageSize=20&js=var%20quote_123%3d{rank:[(x)],pages:(pc)}&token=7bc05d0d4c3c22ef9fca8c2a912d779c&jsName=quote_123&_g=0.6972178580603532";
  20.         urloillist.add(url1);
  21.         urloillist.add(url2);
  22.         for (int i = 0; i < urloillist.size(); i++) {
  23.             //解析url
  24.             oilstocks=ExtMarketOilStockParse.parseurl(urloillist.get(i));
  25.             //存储每页的数据
  26.             MYSQLControl.insertoilStocks(oilstocks);
  27.         }
  28.         //汽车相关股票有6页，对应6个地址
  29.         for (int i = 1; i <6; i++) {
  30.             String urli="http://nufm.dfcfw.com/EM_Finance2014NumericApplication/JS.aspx?type=CT&cmd=C.BK04811&sty=FCOIATA&sortType=C&sortRule=-1&page="+i+"&pageSize=20&js=var%20quote_123%3d{rank:[(x)],pages:(pc)}&token=7bc05d0d4c3c22ef9fca8c2a912d779c&jsName=quote_123&_g=0.23492960370783944";
  31.             urlcarlist.add(urli);
  32.         }
  33.         for (int i = 0; i < urlcarlist.size(); i++) {
  34.             //解析url
  35.             carstocks=ExtMarketOilStockParse.parseurl(urlcarlist.get(i));
  36.             //存储数据
  37.             MYSQLControl.insertcarStocks(carstocks);
  38.         }
  40.     }
  42. }

util

这里有三个文件，HTTPUtils，TimeUtils(这是我自己经常用的一个类，主要是各种日期的转化，如String转化为date，获取当前时间等等)，UumericalUtil(这是一个Float保留几位小数的类)。

  1. package util;
  2. import java.io.BufferedReader;
  3. import java.io.IOException;
  4. import java.io.InputStream;
  5. import java.io.InputStreamReader;
  6. import java.net.URL;
  7. import java.net.URLConnection;
  9. /** * @author:合肥工业大学 管理学院 钱洋 * @email:1563178220@qq.com * @ */
  10. public abstract class HTTPUtils { 
  11.     //这个方法是向后台请求数据，获取html或者json等
  12.     public static String  getRawHtml(String personalUrl) throws InterruptedException,IOException {
  13.         URL url = new URL(personalUrl);
  14.         URLConnection conn = url.openConnection();
  15.         InputStream in=null;
  16.         try {
  17.             conn.setConnectTimeout(3000);
  18.             in = conn.getInputStream();
  19.         } catch (Exception e) {
  20.         }
  21.         //将获取的数据转化为String
  22.         String html = convertStreamToString(in);
  23.         return html;
  24.     }
  25.     //这个方法是将InputStream转化为String
  26.     public static String convertStreamToString(InputStream is) throws IOException {
  27.         if (is == null)
  28.             return "";
  29.         BufferedReader reader = new BufferedReader(new InputStreamReader(is,"utf-8"));
  30.         StringBuilder sb = new StringBuilder();
  31.         String line = null;
  32.         try {
  33.             while ((line = reader.readLine()) != null) {
  34.                 sb.append(line);
  35.             }
  36.         } catch (IOException e) {
  37.             e.printStackTrace();
  38.         } finally {
  39.             try {
  40.                 is.close();
  41.             } catch (IOException e) {
  42.                 e.printStackTrace();
  43.             }
  44.         }
  45.         reader.close();
  46.         return sb.toString();
  48.     }
  49. }

以下类是用来处理各种时间格式之间的转化，大家以后也可以使用。

  1. package util;
  3. import java.text.DateFormat;
  4. import java.text.DecimalFormat;
  5. import java.text.ParseException;
  6. import java.text.SimpleDateFormat;
  7. import java.util.ArrayList;
  8. import java.util.Calendar;
  9. import java.util.Date;
  10. import java.util.List;
  11. /** * @author:合肥工业大学 管理学院 钱洋 * @email:1563178220@qq.com * @ */
  12. public class TimeUtils { 
  14.     public static void main( String[] args ) throws ParseException{
  16.         String time = getMonth("2002-1-08 14:50:38");
  17.         System.out.println(time);
  18.         System.out.println(getDay("2002-1-08 14:50:38"));
  19.         System.out.println(TimeUtils.parseTime("2016-05-19 19:17","yyyy-MM-dd HH:mm"));
  21.     }
  22.     //get current time
  23.     public static String GetNowDate(String formate){  
  24.         String temp_str="";  
  25.         Date dt = new Date();  
  26.         SimpleDateFormat sdf = new SimpleDateFormat(formate);  
  27.         temp_str=sdf.format(dt);  
  28.         return temp_str;  
  29.     }  
  30.     public static String getMonth( String time ){
  32.         SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM");
  33.         Date date = null;
  34.         try {
  36.             date = sdf.parse(time);
  37.             Calendar cal = Calendar.getInstance();
  38.             cal.setTime(date);
  40.         } catch (ParseException e) {
  41.             e.printStackTrace();
  42.         }
  44.         return sdf.format(date);
  46.     }
  48.     public static String getDay( String time ){
  50.         SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd");
  51.         Date date = null;
  52.         try {
  54.             date = sdf.parse(time);
  55.             Calendar cal = Calendar.getInstance();
  56.             cal.setTime(date);
  58.         } catch (ParseException e) {
  59.             e.printStackTrace();
  60.         }
  62.         return sdf.format(date);
  64.     }
  66.     public static Date parseTime(String inputTime) throws ParseException{
  68.         SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");  
  69.         Date date = sdf.parse(inputTime); 
  71.         return date;
  73.     }
  74.     public static String dateToString(Date date, String type) { 
  75.         DateFormat df = new SimpleDateFormat(type);  
  76.         return df.format(date);  
  77.     }
  78.     public static Date parseTime(String inputTime, String timeFormat) throws ParseException{
  80.         SimpleDateFormat sdf = new SimpleDateFormat(timeFormat);  
  81.         Date date = sdf.parse(inputTime); 
  83.         return date;
  85.     }
  87.     public static Calendar parseTimeToCal(String inputTime, String timeFormat) throws ParseException{
  89.         SimpleDateFormat sdf = new SimpleDateFormat(timeFormat);  
  90.         Date date = sdf.parse(inputTime); 
  91.         Calendar calendar = Calendar.getInstance();
  92.         calendar.setTime(date);
  94.         return calendar;
  96.     }
  98.     public static int getDaysBetweenCals(Calendar cal1, Calendar cal2) throws ParseException{
  100.         return (int) ((cal2.getTimeInMillis()-cal1.getTimeInMillis())/(1000*24*3600));
  102.     }
  104.     public static Date parseTime(long inputTime){
  106.         // SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
  107.         Date date= new Date(inputTime);
  108.         return date;
  110.     }
  112.     public static String parseTimeString(long inputTime){
  114.         SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
  115.         Date date= new Date(inputTime);
  116.         return sdf.format(date);
  118.     }
  119.     public static String parseStringTime(String inputTime){
  121.         String date=null;
  122.         try {
  123.             Date date1 = new SimpleDateFormat("yyyyMMddHHmmss").parse(inputTime);
  124.             date=new SimpleDateFormat("yyyy-MM-dd HH:mm:ss").format(date1);
  125.         } catch (ParseException e) {
  126.             // TODO Auto-generated catch block
  127.             e.printStackTrace();
  128.         }
  130.         return date;
  131.     }
  132.     public static List<String> YearMonth(int year) {
  133.         List<String> yearmouthlist=new ArrayList<String>();
  134.         for (int i = 1; i < 13; i++) {
  135.             DecimalFormat dfInt=new DecimalFormat("00");
  136.             String sInt = dfInt.format(i);
  137.             yearmouthlist.add(year+sInt);
  138.         }
  140.         return yearmouthlist;
  141.     } 
  142.     public static List<String> YearMonth(int startyear,int finistyear) {
  143.         List<String> yearmouthlist=new ArrayList<String>();
  144.         for (int i = startyear; i < finistyear+1; i++) {
  145.             for (int j = 1; j < 13; j++) {
  146.                 DecimalFormat dfInt=new DecimalFormat("00");
  147.                 String sInt = dfInt.format(j);
  148.                 yearmouthlist.add(i +"-"+sInt);
  149.             }
  150.         }
  151.         return yearmouthlist;
  152.     } 
  153.     public static List<String> TOAllDay(int year){
  154.         List<String> daylist=new ArrayList<String>();
  155.         SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd"); 
  156.         int m=1;//月份计数 
  157.         while (m<13) 
  158.         { 
  159.             int month=m; 
  160.             Calendar cal=Calendar.getInstance();//获得当前日期对象 
  161.             cal.clear();//清除信息 
  162.             cal.set(Calendar.YEAR,year); 
  163.             cal.set(Calendar.MONTH,month-1);//1月从0开始 
  164.             cal.set(Calendar.DAY_OF_MONTH,1);//设置为1号,当前日期既为本月第一天 
  166.             System.out.println("##########___" + sdf.format(cal.getTime())); 
  168.             int count=cal.getActualMaximum(Calendar.DAY_OF_MONTH); 
  170.             System.out.println("$$$$$$$$$$________" + count); 
  172.             for (int j=0;j<=(count - 2);) 
  173.             { 
  174.                 cal.add(Calendar.DAY_OF_MONTH,+1); 
  175.                 j++; 
  176.                 daylist.add(sdf.format(cal.getTime()));
  177.             } 
  178.             m++; 
  179.         } 
  180.         return daylist;
  181.     }
  182.     //获取昨天的日期
  183.     public static String getyesterday(){
  184.         Calendar   cal   =   Calendar.getInstance();
  185.         cal.add(Calendar.DATE,   -1);
  186.         String yesterday = new SimpleDateFormat( "yyyy-MM-dd ").format(cal.getTime());
  187.         return yesterday;
  188.     }
  189. }

这个类实现的是保留几位小数。如股票价格等，保留两位小数。

  1. package util;
  2. /** * @author:合肥工业大学 管理学院 钱洋 * @email:1563178220@qq.com * @ */
  3. import java.math.BigDecimal;
  4. import java.text.DecimalFormat;
  6. public class UumericalUtil { 
  8.     public static float FloatTO(float f, int number) {
  9.         BigDecimal   b  =   new BigDecimal(f);  
  10.         float   f1   =  b.setScale(number, BigDecimal.ROUND_HALF_UP).floatValue();  
  11.         return f1;  
  12.     }  
  13.     public static String NumberTO(int number) {
  14.         DecimalFormat dfInt=new DecimalFormat("00");
  15.         String sInt = dfInt.format(number);
  16.         System.out.println(sInt);
  17.         return sInt;
  18.     } 
  20. }

parse

parse主要是通过Jsoup或者其他工具来解析html文件。并将解析后的数据，封装在List集合中，将数据通过层层返回到main方法中。如这里只是采用最简单的字符串解析的方式。如下为某一页的数据，这要针对的是此类型的数据进行解析：

  1. var quote_123={rank:["2,002662,京威股份,15.62,0.38,2.49%,2.95,10294,15948185,15.24,15.28,15.65,15.20,-,-,-,-,-,-,-,-,0.00%,0.62,0.17,33.47","2,002536,西泵股份,13.15,0.32,2.49%,3.74,26558,34710121,12.83,12.88,13.27,12.79,-,-,-,-,-,-,-,-,0.00%,0.99,0.87,41.09","1,600741,华域汽车,16.22,0.39,2.46%,2.59,215140,346480560,15.83,15.85,16.26,15.85,-,-,-,-,-,-,-,-,0.12%,1.23,0.75,8.59","1,601689,拓普集团,29.74,0.68,2.34%,3.20,36329,107964394,29.06,29.06,29.94,29.01,-,-,-,-,-,-,-,-,-0.20%,1.34,2.13,34.32","1,603306,华懋科技,33.87,0.74,2.23%,4.50,9251,31242113,33.13,33.14,34.20,32.71,-,-,-,-,-,-,-,-,-0.03%,0.72,1.25,29.60","1,601799,星宇股份,37.40,0.80,2.19%,3.80,5522,20477010,36.60,36.40,37.50,36.11,-,-,-,-,-,-,-,-,0.03%,0.86,0.23,28.43","1,603166,福达股份,14.02,0.29,2.11%,2.91,47265,66170428,13.73,13.80,14.14,13.74,-,-,-,-,-,-,-,-,0.21%,0.96,3.15,95.59","2,002190,成飞集成,32.44,0.66,2.08%,2.99,25213,81219488,31.78,31.63,32.58,31.63,-,-,-,-,-,-,-,-,0.03%,0.86,0.73,93.58","1,600213,亚星客车,14.77,0.30,2.07%,3.46,18878,27820060,14.47,14.52,14.88,14.38,-,-,-,-,-,-,-,-,-0.07%,0.64,0.86,55.39","2,300432,富临精工,21.28,0.43,2.06%,4.70,28707,60945368,20.85,20.60,21.58,20.60,-,-,-,-,-,-,-,-,-0.14%,1.29,2.07,50.58","2,300375,鹏翎股份,21.25,0.42,2.02%,3.94,11367,24164157,20.83,20.83,21.45,20.63,-,-,-,-,-,-,-,-,-0.14%,0.83,1.44,30.27","2,002363,隆基机械,11.47,0.22,1.96%,2.49,33946,38796837,11.25,11.27,11.55,11.27,-,-,-,-,-,-,-,-,0.00%,0.80,0.88,61.45","1,600469,风神股份,11.55,0.22,1.94%,3.09,38444,44305565,11.33,11.33,11.63,11.28,-,-,-,-,-,-,-,-,0.09%,0.67,0.68,27.07","2,002454,松芝股份,12.98,0.24,1.88%,2.83,27839,36056020,12.74,12.70,13.06,12.70,-,-,-,-,-,-,-,-,0.00%,1.17,0.87,25.84","2,002488,金固股份,14.79,0.27,1.86%,2.48,29002,42872475,14.52,14.52,14.88,14.52,-,-,-,-,-,-,-,-,0.00%,0.72,0.75,-","2,002284,亚太股份,13.18,0.24,1.85%,3.32,61756,81198133,12.94,12.87,13.30,12.87,-,-,-,-,-,-,-,-,0.30%,1.10,0.90,58.15","1,603788,宁波高发,35.97,0.64,1.81%,3.40,6719,24160418,35.33,35.21,36.33,35.13,-,-,-,-,-,-,-,-,0.03%,0.59,1.37,34.10","2,000957,中通客车,14.36,0.25,1.77%,2.69,59696,85581415,14.11,14.07,14.45,14.07,-,-,-,-,-,-,-,-,0.00%,0.79,1.25,13.99","2,300304,云意电气,52.12,0.90,1.76%,5.70,179330,922614032,51.22,50.38,52.83,49.91,-,-,-,-,-,-,-,-,-0.04%,1.12,9.35,108.58","2,002607,亚夏汽车,10.03,0.17,1.72%,4.16,27760,27878904,9.86,9.89,10.19,9.78,-,-,-,-,-,-,-,-,-0.30%,0.97,1.03,57.87"],pages:6}
  3. package parse;
  4. /** * @author:合肥工业大学 管理学院 钱洋 * @email:1563178220@qq.com * @ */
  5. import java.util.ArrayList;
  6. import java.util.List;
  7. import model.ExtMarketOilStockModel;
  8. import util.HTTPUtils;
  9. import util.TimeUtils;
  10. import util.UumericalUtil;
  11. public class ExtMarketOilStockParse {
  12.     public static List<ExtMarketOilStockModel> parseurl(String url) throws Exception {
  13.         List<ExtMarketOilStockModel> list=new ArrayList<ExtMarketOilStockModel>();
  14.         String response=HTTPUtils.getRawHtml(url);
  15.         String html = response.toString();
  16.         String jsonarra=html.split("rank:")[1].split(",pages")[0];
  17.         String stocks[]=jsonarra.split("\",");
  18.         List<String> stocklist=new ArrayList<String>();
  19.         for (int i = 0; i < stocks.length; i++) { 
  20.             stocklist.add(stocks[i].replace("[\"", "").replace("\"", "").replace("]", ""));
  21.             System.out.println(stocks[i].replace("[\"", "").replace("\"", "").replace("]", ""));
  22.         }
  23.         for (int i = 0; i < stocklist.size(); i++) { 
  24.             String date=TimeUtils.GetNowDate("yyyy-MM-dd");
  25.             String stock_id=stocklist.get(i).split(",")[1];
  26.             String stock_name=stocklist.get(i).split(",")[2];
  27.             float stock_price=0;
  28.             float stock_change=0;
  29.             float stock_range=0;
  30.             float stock_amplitude=0;
  31.             int stock_trading_number=0;
  32.             int stock_trading_value=0;
  33.             float stock_yesterdayfinish_price=0;
  34.             float stock_todaystart_price=0;
  35.             float stock_max_price=0;
  36.             float stock_min_price=0;
  37.             float stock_fiveminuate_change=0;
  38.             if (!stocklist.get(i).split(",")[3].equals("-")) {
  39.                 //价格
  40.                 stock_price=Float.parseFloat(stocklist.get(i).split(",")[3]);
  41.                 //涨跌额
  42.                 stock_change=Float.parseFloat(stocklist.get(i).split(",")[4]);
  43.                 System.out.println(stock_change);
  44.                 //涨跌幅
  45.                 stock_range=UumericalUtil.FloatTO((float) (Float.parseFloat(stocklist.get(i).split(",")[5].replace("%", ""))*0.01),4);
  46.                 stock_amplitude=UumericalUtil.FloatTO((float) (Float.parseFloat(stocklist.get(i).split(",")[6].replace("%", ""))*0.01),4);;
  47.                 stock_trading_number=Integer.parseInt(stocklist.get(i).split(",")[7].replace("%", ""));
  48.                 stock_trading_value=Integer.parseInt(stocklist.get(i).split(",")[8].replace("%", ""));
  49.                 stock_yesterdayfinish_price=Float.parseFloat(stocklist.get(i).split(",")[9]);
  50.                 stock_todaystart_price=Float.parseFloat(stocklist.get(i).split(",")[10]);
  51.                 stock_max_price=Float.parseFloat(stocklist.get(i).split(",")[11]);
  52.                 stock_min_price=Float.parseFloat(stocklist.get(i).split(",")[12]);
  53.                 stock_fiveminuate_change=UumericalUtil.FloatTO((float) (Float.parseFloat(stocklist.get(i).split(",")[21].replace("%", ""))*0.01),4);;
  54.                 System.out.println(stock_fiveminuate_change);
  55.             }
  56.             String craw_time=TimeUtils.GetNowDate("yyyy-MM-dd HH:mm:ss");
  57.             ExtMarketOilStockModel model=new ExtMarketOilStockModel();
  58.             model.setDate(date);
  59.             model.setStock_id(stock_id);
  60.             model.setStock_name(stock_name);
  61.             model.setStock_price(stock_price);
  62.             model.setStock_change(stock_change);
  63.             model.setStock_range(stock_range);
  64.             model.setStock_amplitude(stock_amplitude);
  65.             model.setStock_trading_number(stock_trading_number);
  66.             model.setStock_trading_value(stock_trading_value);
  67.             model.setStock_yesterdayfinish_price(stock_yesterdayfinish_price);
  68.             model.setStock_todaystart_price(stock_todaystart_price);
  69.             model.setStock_max_price(stock_max_price);
  70.             model.setStock_min_price(stock_min_price);
  71.             model.setStock_fiveminuate_change(stock_fiveminuate_change);
  72.             model.setCraw_time(craw_time);
  73.             list.add(model);
  74.         }
  75.         return list; 
  76.     }
  77. }

db

db中包含两个java文件，MyDataSource，MYSQLControl。这两个文件的作用已在前面说明了。

  1. package db;
  2. /** * @author:合肥工业大学 管理学院 钱洋 * @email:1563178220@qq.com * @ */
  3. import javax.sql.DataSource;
  4. import org.apache.commons.dbcp2.BasicDataSource;
  6. public class MyDataSource { 
  8.     public static DataSource getDataSource(String connectURI){
  10.         BasicDataSource ds = new BasicDataSource();
  11.          //MySQL的jdbc驱动
  12.         ds.setDriverClassName("com.mysql.jdbc.Driver");
  13.         ds.setUsername("root");              //所要连接的数据库名
  14.         ds.setPassword("112233");                //MySQL的登陆密码
  15.         ds.setUrl(connectURI);
  17.         return ds;
  19.     }
  21. }
  23. package db;
  24. import java.sql.SQLException;
  25. import java.util.List;
  26. import javax.sql.DataSource;
  27. import org.apache.commons.dbutils.QueryRunner;
  28. import org.apache.commons.dbutils.ResultSetHandler;
  29. import org.apache.commons.dbutils.handlers.BeanListHandler;
  30. import org.apache.commons.dbutils.handlers.ColumnListHandler;
  31. import org.apache.commons.dbutils.handlers.ScalarHandler;
  32. import org.apache.commons.logging.Log;
  33. import org.apache.commons.logging.LogFactory;
  34. import model.ExtMarketOilStockModel;
  35. /** * @author:合肥工业大学 管理学院 钱洋 * @email:1563178220@qq.com * @ */
  36. public class MYSQLControl {
  37.     static final Log logger = LogFactory.getLog(MYSQLControl.class);
  38.     static DataSource ds = MyDataSource.getDataSource("jdbc:mysql://127.0.0.1:3306/datacollection");
  39.     static QueryRunner qr = new QueryRunner(ds);
  40.     //第一类方法
  41.     public static void executeUpdate(String sql){
  42.         try {
  43.             qr.update(sql);
  44.         } catch (SQLException e) {
  45.             logger.error(e);
  46.         }
  47.     }
  48.     //按照SQL查询单个结果
  49.     public static Object getScalaBySQL ( String sql ){
  51.         ResultSetHandler<Object> h = new ScalarHandler<Object>(1);
  52.         Object obj = null;
  53.         try {
  54.             obj = qr.query(sql, h);
  55.         } catch (SQLException e) {
  56.             e.printStackTrace();
  57.         }
  58.         return obj;
  60.     }
  61.     //按照SQL查询多个结果
  62.     public static <T> List<T> getListInfoBySQL (String sql, Class<T> type ){
  63.         List<T> list = null;
  64.         try {
  65.             list = qr.query(sql,new BeanListHandler<T>(type));
  66.         } catch (SQLException e) {
  67.             e.printStackTrace();
  68.         }
  69.         return list;
  70.     }
  71.     //查询一列
  72.     public static List<Object> getListOneBySQL (String sql,String id){
  73.         List<Object> list=null;
  75.         try {
  76.             list = (List<Object>) qr.query(sql, new ColumnListHandler(id));
  77.         } catch (SQLException e) {
  78.             e.printStackTrace();
  79.         }
  80.         return list;
  81.     }
  82.     //此种数据库操作方法需要优化
  83.     public static int insertoilStocks ( List<ExtMarketOilStockModel> oilstocks ) {
  85.         Object[][] params = new Object[oilstocks.size()][17];
  86.         int c = 0;  //success number of update
  87.         int[] sum;
  88.         for ( int i = 0; i < oilstocks.size(); i++ ){
  89.             params[i][0] = oilstocks.get(i).getDate();
  90.             params[i][1] = oilstocks.get(i).getStock_id();
  91.             params[i][2] = oilstocks.get(i).getStock_name();
  92.             params[i][3] = oilstocks.get(i).getStock_price();
  93.             params[i][4] = oilstocks.get(i).getStock_change();
  94.             params[i][5] = oilstocks.get(i).getStock_range();
  95.             params[i][6] = oilstocks.get(i).getStock_amplitude();
  96.             params[i][7] = oilstocks.get(i).getStock_trading_number();
  97.             params[i][8] = oilstocks.get(i).getStock_trading_value();
  98.             params[i][9] = oilstocks.get(i).getStock_yesterdayfinish_price();
  99.             params[i][10] = oilstocks.get(i).getStock_todaystart_price();
  100.             params[i][11] = oilstocks.get(i).getStock_max_price();
  101.             params[i][12] = oilstocks.get(i).getStock_min_price();
  102.             params[i][13] = oilstocks.get(i).getStock_fiveminuate_change();
  103.             params[i][14] = oilstocks.get(i).getCraw_time();
  104.             params[i][15] = null;
  105.             params[i][16] = null;
  106.         }
  108.         QueryRunner qr = new QueryRunner(ds);
  109.         try {
  110.             sum = qr.batch("INSERT INTO `datacollection`.`ext_market_oil_stock` VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)", params);
  111.         } catch (SQLException e) {
  112.             System.out.println(e);
  113.         }
  114.         System.out.println("石油数据入库完毕");
  116.         return c;
  118.     }
  119.     //此种数据库操作方法需要优化
  120.     public static int insertcarStocks ( List<ExtMarketOilStockModel> carstocks ) {
  122.         int c = 0;  //success number of update
  123.         int[] sum;
  124.         Object[][] params1 = new Object[carstocks.size()][17];
  125.         int c1 = 0; //success number of update
  126.         for ( int i = 0; i < carstocks.size(); i++ ){
  127.             params1[i][0] = carstocks.get(i).getDate();
  128.             params1[i][1] = carstocks.get(i).getStock_id();
  129.             params1[i][2] = carstocks.get(i).getStock_name();
  130.             params1[i][3] = carstocks.get(i).getStock_price();
  131.             params1[i][4] = carstocks.get(i).getStock_change();
  132.             params1[i][5] = carstocks.get(i).getStock_range();
  133.             params1[i][6] = carstocks.get(i).getStock_amplitude();
  134.             params1[i][7] = carstocks.get(i).getStock_trading_number();
  135.             params1[i][8] = carstocks.get(i).getStock_trading_value();
  136.             params1[i][9] = carstocks.get(i).getStock_yesterdayfinish_price();
  137.             params1[i][10] = carstocks.get(i).getStock_todaystart_price();
  138.             params1[i][11] = carstocks.get(i).getStock_max_price();
  139.             params1[i][12] = carstocks.get(i).getStock_min_price();
  140.             params1[i][13] = carstocks.get(i).getStock_fiveminuate_change();
  141.             params1[i][14] = carstocks.get(i).getCraw_time();
  142.             params1[i][15] = null;
  143.             params1[i][16] = null;
  144.         }
  145.         QueryRunner qr = new QueryRunner(ds);
  146.         try {
  147.         //插入的数据表及数据
  148.             sum = qr.batch("INSERT INTO `datacollection`.`ext_market_car_stock` VALUES (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)", params1);
  150.         } catch (SQLException e) {
  151.             System.out.println(e);
  152.         }
  153.         System.out.println("汽车数据入库完毕");
  155.         return c;
  157.     }
  159. }

这样按道理整个爬虫，程序就写完了，运行main方法就行了。如下图，为main方法获取数据的部分结果。

这里写图片描述

问题所在

问题1：针对股票这种数据，每周1到周五都会发布相关股票数据，那么如何每天定时定点让程序自动的去抓取，而不是手工每天运行一下呢？

问题二：股票节假日，是不会开盘的，当网页中存在此数据，即网页中的显示，没有时间标签。针对此，又该如何处理呢？

首先，我带大家来看看我的数据库设计。

这里写图片描述

解决方法

这里使用Quartz实线定期运行程序，即上面提的第一个问题。(http://blog.csdn.net/qy20115549/article/details/52723907)。
针对第二个问题使用是：即如何判断当天股票不开盘，采用的方法是从数据库中随机抽取三个股票(上次时间的，如今天是1月21日，周六，随机从数据库中抽取1月20日的三只股票。将1月20日的三只股票与今天相同id的股票价格进行比较，如果三个股票的价格都相同，则判断，改天为节假日，股票价格没有变动，无需将数据插入数据库)。

job

  1. package job;
  3. import java.util.ArrayList;
  4. import java.util.List;
  5. import org.quartz.Job; 
  6. import org.quartz.JobExecutionContext; 
  7. import org.quartz.JobExecutionException;
  8. import db.MYSQLControl;
  9. import model.ExtMarketOilStockModel;
  10. import parse.ExtMarketOilStockParse;
  11. import timecontrol.TimeControl;
  12.  /** * @author:合肥工业大学 管理学院 钱洋 * @email:1563178220@qq.com * @ */
  13. public class ExtMarketOilStockJob implements Job {  
  15.     @Override 
  16.     public void execute(JobExecutionContext arg0) throws JobExecutionException {
  17.         //获取上次的插入股票日期，加入判断是否为节假日
  18.         List<ExtMarketOilStockModel> randomlist = MYSQLControl.getListInfoBySQL("select stock_id,stock_price,stock_change from ext_market_oil_stock where date = (select date from ext_market_oil_stock order by date desc limit 1) ",ExtMarketOilStockModel.class);
  19.         //表格更新时间
  21.         List<String> urloillist=new ArrayList<String>();
  22.         List<String> urlcarlist=new ArrayList<String>();
  23.         List<ExtMarketOilStockModel> oilstocks=new ArrayList<ExtMarketOilStockModel>();
  24.         List<ExtMarketOilStockModel> carstocks=new ArrayList<ExtMarketOilStockModel>();
  25.         String url1="http://nufm.dfcfw.com/EM_Finance2014NumericApplication/JS.aspx?type=CT&cmd=C.BK04641&sty=FCOIATA&sortType=C&sortRule=-1&page=1&pageSize=20&js=var%20quote_123%3d{rank:[(x)],pages:(pc)}&token=7bc05d0d4c3c22ef9fca8c2a912d779c&jsName=quote_123&_g=0.13204790262127375";
  26.         String url2="http://nufm.dfcfw.com/EM_Finance2014NumericApplication/JS.aspx?type=CT&cmd=C.BK04641&sty=FCOIATA&sortType=C&sortRule=-1&page=2&pageSize=20&js=var%20quote_123%3d{rank:[(x)],pages:(pc)}&token=7bc05d0d4c3c22ef9fca8c2a912d779c&jsName=quote_123&_g=0.6972178580603532";
  27.         urloillist.add(url1);
  28.         urloillist.add(url2);
  29.         int judge=0;
  30.         for (int i = 0; i < urloillist.size(); i++) {
  31.             try {
  32.                 oilstocks=ExtMarketOilStockParse.parseurl(urloillist.get(i));
  33.             } catch (Exception e) {
  34.                 e.printStackTrace();
  35.             }
  37.             for (int j = 0; j < oilstocks.size(); j++) {
  38.                 String stock_id=oilstocks.get(j).getStock_id();
  39.                 float stock_price=oilstocks.get(j).getStock_price();
  40.                 if (stock_id.equals(randomlist.get(0).getStock_id())) {
  41.                     if (stock_price==randomlist.get(0).getStock_price()) {
  42.                         judge++;
  43.                     }
  44.                 }
  45.             }
  46.             for (int j = 0; j < oilstocks.size(); j++) {
  47.                 String stock_id=oilstocks.get(j).getStock_id();
  48.                 float stock_price=oilstocks.get(j).getStock_price();
  49.                 if (stock_id.equals(randomlist.get(1).getStock_id())) {
  50.                     if (stock_price==randomlist.get(1).getStock_price()) {
  51.                         judge++;
  52.                     }
  53.                 }
  54.             }
  55.             for (int j = 0; j < oilstocks.size(); j++) {
  56.                 String stock_id=oilstocks.get(j).getStock_id();
  57.                 float stock_price=oilstocks.get(j).getStock_price();
  58.                 if (stock_id.equals(randomlist.get(2).getStock_id())) {
  59.                     if (stock_price==randomlist.get(2).getStock_price()) {
  60.                         judge++;
  61.                     }
  62.                 }
  63.             }
  64.             if (judge!=3) {
  65.                 MYSQLControl.insertoilStocks(oilstocks);
  66.             }
  67.         }
  68.         if (judge!=3) {
  69.             for (int i = 1; i <6; i++) {
  70.                 String urli="http://nufm.dfcfw.com/EM_Finance2014NumericApplication/JS.aspx?type=CT&cmd=C.BK04811&sty=FCOIATA&sortType=C&sortRule=-1&page="+i+"&pageSize=20&js=var%20quote_123%3d{rank:[(x)],pages:(pc)}&token=7bc05d0d4c3c22ef9fca8c2a912d779c&jsName=quote_123&_g=0.23492960370783944";
  71.                 urlcarlist.add(urli);
  72.             }
  73.             for (int i = 0; i < urlcarlist.size(); i++) {
  74.                 try {
  75.                     carstocks=ExtMarketOilStockParse.parseurl(urlcarlist.get(i));
  76.                 } catch (Exception e) {
  77.                     e.printStackTrace();
  78.                 }
  79.                 MYSQLControl.insertcarStocks(carstocks);
  80.             }
  81.         }
  83.     } 
  85. }

jobmain

如下，控制的时间是每周一到周五，8点39执行job，即每天都去抓取数据。

  1. package jobmain;
  2. import static org.quartz.CronScheduleBuilder.cronSchedule;
  3. import static org.quartz.JobBuilder.newJob;
  4. import static org.quartz.TriggerBuilder.newTrigger;
  5. import java.text.SimpleDateFormat;
  6. import java.util.Date;
  7. import org.quartz.CronTrigger;
  8. import org.quartz.JobDetail;
  9. import org.quartz.Scheduler;
  10. import org.quartz.SchedulerFactory;
  11. import org.quartz.impl.StdSchedulerFactory;
  12. import job.ExtMarketOilStockJob;
  13.  /** * @author:合肥工业大学 管理学院 钱洋 * @email:1563178220@qq.com * @ */
  14. public class ExtMarketOilStockJobMain { 
  16.     public void go() throws Exception { 
  17.         // 首先，必需要取得一个Scheduler的引用 
  18.         SchedulerFactory sf = new StdSchedulerFactory(); 
  19.         Scheduler sched = sf.getScheduler(); 
  20.         //jobs可以在scheduled的sched.start()方法前被调用 
  21.         JobDetail job = newJob(ExtMarketOilStockJob.class).withIdentity("stockjob", "stockgroup").build(); 
  22.         //每周一到周五8点39开始执行job
  23.         CronTrigger trigger = newTrigger().withIdentity("stocktrigger", "stockgroup").withSchedule(cronSchedule("0 39 20 ? * MON-FRI")).build(); 
  24.         Date ft = sched.scheduleJob(job, trigger); 
  25.         SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss SSS"); 
  26.         System.out.println(job.getKey() + " 已被安排执行于: " + sdf.format(ft) + "，并且以如下重复规则重复执行: " + trigger.getCronExpression()); 
  27.         sched.start(); 
  28.     } 
  29.     public static void main(String[] args) throws Exception { 
  30.         ExtMarketOilStockJobMain maingo = new ExtMarketOilStockJobMain(); 
  31.         maingo.go(); 
  32.     } 
  34. }

运行jobmain中的类，便可以实现每天定点爬取数据。

发表评论

表情：
评论列表 （有 0 条评论，756人围观）

还没有评论，来说两句吧...

相关阅读

    相关 Python如何爬取免费爬虫ip

    做过大数据抓取的程序员应该都知道，正常市面上的爬虫ip只分为两种，一种是API提取式的，还有一种是账密形式隧道模式的。往往因为高昂费用而止步。对于初学者觉得没有必要，我们...

    桃扇骨桃扇骨/ 0 赞/ 169 阅读