log4j按照日期或时间格式生成备份日志文件

╰+哭是因爲堅強的太久メ 2022-05-18 00:43 319阅读 0赞

**1.写在前面：**  
log4j使用的是log4j 1.2.x。  
配置文件使用的是log4j.xml配置文件格式，没有使用log4j.properties文件形式，但是基本差不多。

**2.需求**  
多线程，多节点，每天一个模块要产生2-3GB的日志，分散到8个节点，每个节点差不多400-500MB。打开和查询非常麻烦，而且不能够使用ctrl+f全文搜索，每次使用全文搜索，CPU占用率都会达到85+%。所以需要将日志文件按照小时切割。

**3.遇到的坑**

**坑1**

<?xml version="1.0" encoding="UTF-8"?>
    <!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">
    <log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/">
        <appender name="ConsoleAppender" class="org.apache.log4j.ConsoleAppender">
            
            <layout class="org.apache.log4j.PatternLayout">
                <param name="ConversionPattern" value="%d{yyyy-MM-dd HH:mm:ss.SSS} [%t] [%-5p] %c - %m%n" />
            </layout>
            
            <filter class="org.apache.log4j.varia.LevelRangeFilter">
                <param name="levelMin" value="INFO" />
                <param name="levelMax" value="ERROR" />
                <param name="AcceptOnMatch" value="true" />
            </filter>
        </appender>
    
        <appender name="RollingFileAppender" class="org.apache.log4j.DailyRollingFileAppender">
            <param name="DatePattern" value="'.'yyyy-MM-dd-HH'.log'" />
            <param name="Encoding" value="UTF-8" />
            <param name="File" value="${catalina.home}/logs/txbd.log" />
            <param name="Append" value="true" />
            <param name="Threshold" value="DEBUG" />
            <layout class="org.apache.log4j.PatternLayout">
                <param name="ConversionPattern" value="%d{yyyy-MM-dd HH:mm:ss.SSS} [%t] [%-5p] %c - %m%n" />
            </layout>
        </appender>
    
      <root> 
        <priority value ="DEBUG" /> 
        <appender-ref ref="ConsoleAppender" />
        <appender-ref ref="RollingFileAppender" />
      </root>
    </log4j:configuration>

理论上产生的结果：当前的日志名为txbd.log，等到下一个小时后会产生一个备份的日志文件txbd.log2018-01-01.log文件。  
出现了LOG4J:ERROR Failed to rename的错误。  
查看源码后发现源码的内容：

File file = new File(fileName);
    boolean result = file.renameTo(target);
    if(result)
        LogLog.debug(fileName + " -> " + scheduledFilename);
    else
        LogLog.error("Failed to rename [" + fileName + "] to [" + scheduledFilename + "].");
    try{
        setFile(fileName, true, bufferedIO, bufferSize);
    }catch(IOException e){
        errorHandler.error("setFile(" + fileName + ", true) call failed.");
    }

`这个原因在于日志文件被线程占用，rename操作服务进行，所以会提示这个错误`

解决办法：看了网上的解决办法，清一色的是将rename改成copy方法。新创建一个文件，然后将上一小时的日志拷贝到这个文件中？试一下。  
**坑2**

`自定义DailyRollingFileAppender类，将rename方法改成copy方法`

/** * log4j日志拓展类 * 将新文件写入到txbd.log * 旧文件写入到txbd.logyyyy-MM-dd-HH.log文件中 * 修复无法rename的bug，但是出现了写入后又被清理掉的问题（待解决） * @author T */
    public class DailyRollingFileAppenderX1 extends FileAppender { 
    
          static final int TOP_OF_TROUBLE=-1;
          static final int TOP_OF_MINUTE = 0;
          static final int TOP_OF_HOUR   = 1;
          static final int HALF_DAY      = 2;
          static final int TOP_OF_DAY    = 3;
          static final int TOP_OF_WEEK   = 4;
          static final int TOP_OF_MONTH  = 5;
    
          private String datePattern = "'.'yyyy-MM-dd";
    
          private String scheduledFilename;
    
          private long nextCheck = System.currentTimeMillis () - 1;
    
          Date now = new Date();
    
          SimpleDateFormat sdf;
    
          RollingCalendar rc = new RollingCalendar();
    
          int checkPeriod = TOP_OF_TROUBLE;
    
          // The gmtTimeZone is used only in computeCheckPeriod() method.
          static final TimeZone gmtTimeZone = TimeZone.getTimeZone("GMT");
    
          public DailyRollingFileAppenderX1() {
    
          }
    
          public DailyRollingFileAppenderX1 (Layout layout, String filename, String datePattern) throws IOException {
            super(layout, filename, true);
            this.datePattern = datePattern;
            activateOptions();
          }
    
          public void setDatePattern(String pattern) {
            datePattern = pattern;
          }
    
          public String getDatePattern() {
            return datePattern;
          }
    
          public void activateOptions() {
            super.activateOptions();
            if(datePattern != null && fileName != null) {
              now.setTime(System.currentTimeMillis());
              sdf = new SimpleDateFormat(datePattern);
              int type = computeCheckPeriod();
              printPeriodicity(type);
              rc.setType(type);
              File file = new File(fileName);
              scheduledFilename = fileName+sdf.format(new Date(file.lastModified()));
    
            } else {
              LogLog.error("Either File or DatePattern options are not set for appender ["
                   +name+"].");
            }
          }
    
          void printPeriodicity(int type) {
            switch(type) {
            case TOP_OF_MINUTE:
              LogLog.debug("Appender ["+name+"] to be rolled every minute.");
              break;
            case TOP_OF_HOUR:
              LogLog.debug("Appender ["+name +"] to be rolled on top of every hour.");
              break;
            case HALF_DAY:
              LogLog.debug("Appender ["+name +"] to be rolled at midday and midnight.");
              break;
            case TOP_OF_DAY:
              LogLog.debug("Appender ["+name +"] to be rolled at midnight.");
              break;
            case TOP_OF_WEEK:
              LogLog.debug("Appender ["+name +"] to be rolled at start of week.");
              break;
            case TOP_OF_MONTH:
              LogLog.debug("Appender ["+name +"] to be rolled at start of every month.");
              break;
            default:
              LogLog.warn("Unknown periodicity for appender ["+name+"].");
            }
          }
    
          int computeCheckPeriod() {
            RollingCalendar rollingCalendar = new RollingCalendar(gmtTimeZone, Locale.getDefault());
            // set sate to 1970-01-01 00:00:00 GMT
            Date epoch = new Date(0);
            if(datePattern != null) {
              for(int i = TOP_OF_MINUTE; i <= TOP_OF_MONTH; i++) {
            SimpleDateFormat simpleDateFormat = new SimpleDateFormat(datePattern);
            simpleDateFormat.setTimeZone(gmtTimeZone); // do all date formatting in GMT
            String r0 = simpleDateFormat.format(epoch);
            rollingCalendar.setType(i);
            Date next = new Date(rollingCalendar.getNextCheckMillis(epoch));
            String r1 =  simpleDateFormat.format(next);
            //System.out.println("Type = "+i+", r0 = "+r0+", r1 = "+r1);
            if(r0 != null && r1 != null && !r0.equals(r1)) {
              return i;
            }
              }
            }
            return TOP_OF_TROUBLE; // Deliberately head for trouble...
          }
    
          void rollOver() throws IOException {
    
            if (datePattern == null) {
              errorHandler.error("Missing DatePattern option in rollOver().");
              return;
            }
    
            String datedFilename = fileName+sdf.format(now);
    
            if (scheduledFilename.equals(datedFilename)) {
              return;
            }
    
            this.closeFile();
    
            File target  = new File(scheduledFilename);
            if (target.exists()) {
              target.delete();
            }
    
    
            File file = new File(fileName);  
            boolean result = copy(file, target);
            System.out.println("=====>进行日志文件拆分："+fileName);
            if (result) {  
                //增加清空日志的方法，在复制成功后，清空原来的日志
                FileWriter fw =  new FileWriter(file);  
                fw.write("");  
                fw.flush();  
                fw.close();
                LogLog.debug(fileName + " -> " + scheduledFilename);  
            } else {
                System.out.println("=====>日志文件拆分失败");
                LogLog.error("Failed to rename [" + fileName + "] to [" + scheduledFilename + "].");  
            }
    
    
            try {
    
              this.setFile(fileName, true, this.bufferedIO, this.bufferSize);
            }
            catch(IOException e) {
              errorHandler.error("setFile("+fileName+", true) call failed.");
            }
            scheduledFilename = datedFilename;
          }
    
          /** * 日志文件复制 * @param src 源文件名 * @param dst 新文件名 * @return * @throws IOException */
          private boolean copy(File src, File dst) throws IOException {
                try {
                    InputStream in = new FileInputStream(src);
                    OutputStream out = new FileOutputStream(dst);
    
                    // Transfer bytes from in to out
                    byte[] buf = new byte[8192];
                    int len;
                    while ((len = in.read(buf)) > 0) {
                        out.write(buf, 0, len);
                    }
                    in.close();
                    out.close();
                    return true;
                } catch (FileNotFoundException e) {
                    LogLog.error("源文件不存在,或者目标文件无法被识别." );
                    return false;
                } catch (IOException e) {
                    LogLog.error("文件读写错误.");
                    return false;
                }
            }
    
          private boolean copy2(File file1, File file2) throws IOException{
                // 3.将创建的节点流的对象作为形参传递给缓冲流的构造器中
                BufferedInputStream bis = null;
                BufferedOutputStream bos = null;
                try{
                    // 2.创建相应的节点流
                    FileInputStream fis = new FileInputStream(file1);
                    FileOutputStream fos = new FileOutputStream(file2);
    
                    bis = new BufferedInputStream(fis);
                    bos = new BufferedOutputStream(fos);
                    // 4.实现文件的复制
                    byte[] b = new byte[4096];// 每次运20个，可按照实际文件大小调整
                    int len;
                    while ((len = bis.read(b)) != -1){
                        bos.write(b, 0, len);
                    }
                    return true;
                } catch (Exception e) {
                    e.printStackTrace();
                    return false;
                } finally {
                    // 关闭相应的流
                    if (bos != null) {
                        try {
                            bos.close();
                        } catch (IOException e) {
                            e.printStackTrace();
                        } finally {
                            if (bis != null){
                                try {
                                    bis.close();
                                } catch (IOException e) {
                                    e.printStackTrace();
                                }
                            }
                        }
                    }
                }
            }
    
    
          protected void subAppend(LoggingEvent event) {
            long n = System.currentTimeMillis();
            if (n >= nextCheck) {
              now.setTime(n);
              nextCheck = rc.getNextCheckMillis(now);
              try {
            rollOver();
              }
              catch(IOException ioe) {
                  if (ioe instanceof InterruptedIOException) {
                      Thread.currentThread().interrupt();
                  }
                  LogLog.error("rollOver() failed.", ioe);
              }
            }
            super.subAppend(event);
           }
        }
    
        class RollingCalendar1 extends GregorianCalendar {
          private static final long serialVersionUID = -3560331770601814177L;
    
          int type = DailyRollingFileAppenderX1.TOP_OF_TROUBLE;
    
          RollingCalendar1() {
            super();
          }  
    
          RollingCalendar1(TimeZone tz, Locale locale) {
            super(tz, locale);
          }  
    
          void setType(int type) {
            this.type = type;
          }
    
          public long getNextCheckMillis(Date now) {
            return getNextCheckDate(now).getTime();
          }
    
          public Date getNextCheckDate(Date now) {
            this.setTime(now);
    
            switch(type) {
            case DailyRollingFileAppenderX1.TOP_OF_MINUTE:
            this.set(Calendar.SECOND, 0);
            this.set(Calendar.MILLISECOND, 0);
            this.add(Calendar.MINUTE, 1);
            break;
            case DailyRollingFileAppenderX1.TOP_OF_HOUR:
            this.set(Calendar.MINUTE, 0);
            this.set(Calendar.SECOND, 0);
            this.set(Calendar.MILLISECOND, 0);
            this.add(Calendar.HOUR_OF_DAY, 1);
            break;
            case DailyRollingFileAppenderX1.HALF_DAY:
            this.set(Calendar.MINUTE, 0);
            this.set(Calendar.SECOND, 0);
            this.set(Calendar.MILLISECOND, 0);
            int hour = get(Calendar.HOUR_OF_DAY);
            if(hour < 12) {
              this.set(Calendar.HOUR_OF_DAY, 12);
            } else {
              this.set(Calendar.HOUR_OF_DAY, 0);
              this.add(Calendar.DAY_OF_MONTH, 1);
            }
            break;
            case DailyRollingFileAppenderX1.TOP_OF_DAY:
            this.set(Calendar.HOUR_OF_DAY, 0);
            this.set(Calendar.MINUTE, 0);
            this.set(Calendar.SECOND, 0);
            this.set(Calendar.MILLISECOND, 0);
            this.add(Calendar.DATE, 1);
            break;
            case DailyRollingFileAppenderX1.TOP_OF_WEEK:
            this.set(Calendar.DAY_OF_WEEK, getFirstDayOfWeek());
            this.set(Calendar.HOUR_OF_DAY, 0);
            this.set(Calendar.MINUTE, 0);
            this.set(Calendar.SECOND, 0);
            this.set(Calendar.MILLISECOND, 0);
            this.add(Calendar.WEEK_OF_YEAR, 1);
            break;
            case DailyRollingFileAppenderX1.TOP_OF_MONTH:
            this.set(Calendar.DATE, 1);
            this.set(Calendar.HOUR_OF_DAY, 0);
            this.set(Calendar.MINUTE, 0);
            this.set(Calendar.SECOND, 0);
            this.set(Calendar.MILLISECOND, 0);
            this.add(Calendar.MONTH, 1);
            break;
            default:
            throw new IllegalStateException("Unknown periodicity type.");
            }
            return getTime();
          }
    
        }

写了两个copy方法，效果是一样的。  
感觉应该没问题哈！

结果一测试，发现文件都生成了，但是备份的日志文件不完整，确切说就是只备份了一点点的日志信息。  
经过观察发现，日志确实是全部写入到了备份的日志文件中。但是，写完之后，备份文件立马就被清理掉了！  
这个问题是很奇怪，怀疑是：

//增加清空日志的方法，在复制成功后，清空原来的日志
    FileWriter fw =  new FileWriter(file);  
    fw.write("");  
    fw.flush();  
    fw.close();

这段程序有问题，导致日志被清理掉。  
后来将这部分注释掉，确实日志都备份的比较完整。但是又引入了一个新的问题：日志重复。  
所以这段肯定是不能注释的。

猜测应该是流没有刷新的问题，未进行下去

`对于网上那些不经过实验，就乱转发文章的作者表示无奈`

**4.找到了自己想要的方案**

前面遇到了问题，都行不通，那么可不可以换一个角度想想呢：

`创建新文件，将日志写入到新文件中`

/** * 日志拓展类，按照小时为单位，每小时会生成一个新文件，并将接下来的一小时的日志写入到改文件中 * @author T */
    public class DailyRollingFileAppenderX extends RollingFileAppender { 
        static final int TOP_OF_TROUBLE = -1;
        static final int TOP_OF_MINUTE = 0;
        static final int TOP_OF_HOUR = 1;
        static final int HALF_DAY = 2;
        static final int TOP_OF_DAY = 3;
        static final int TOP_OF_WEEK = 4;
        static final int TOP_OF_MONTH = 5;
        private String datePattern = "'.'yyyy-MM-dd-HH";
        private String scheduledFilename;
        private long nextCheck = System.currentTimeMillis() - 1L;
        private String val;
        Date now = new Date();
        SimpleDateFormat sdf;
        RollingCalendar rc = new RollingCalendar();
    
        int checkPeriod = -1;
    
        static final TimeZone gmtTimeZone = TimeZone.getTimeZone("GMT");
    
        public DailyRollingFileAppenderX() {
    
        }
    
        public DailyRollingFileAppenderX(Layout layout, String filename) throws IOException {
    
            super(layout, filename);
            //System.out.println("------------------");
        }
    
        public DailyRollingFileAppenderX(Layout layout, String filename, String datePattern) throws IOException {
            super(layout, filename, true);
            // System.out.println(new SimpleDateFormat("'"+filename+"'yyyy-MM-dd'.log'").format(new Date()));
            this.datePattern = datePattern;
            activateOptions();
        }
    
        public void setDatePattern(String pattern) {
            this.datePattern = pattern;
        }
    
        public String getDatePattern() {
            return this.datePattern;
        }
    
        public void setFile(String file) {
            // Trim spaces from both ends. The users probably does not want
            // trailing spaces in file names.
            this.val = file.trim();
            SimpleDateFormat formats = new SimpleDateFormat("'" + val + "'yyyy-MM-dd-HH'.log'");
            String fileNameNew = formats.format(new Date());
            fileName = fileNameNew;
        }
    
        public void activateOptions() {
            super.activateOptions();
            if ((this.datePattern != null) && (this.fileName != null)) {
                this.now.setTime(System.currentTimeMillis());
                this.sdf = new SimpleDateFormat(this.datePattern);
                int type = computeCheckPeriod();
                printPeriodicity(type);
                this.rc.setType(type);
    
                File file = new File(fileName);
                this.scheduledFilename = (this.fileName + this.sdf.format(new Date(file.lastModified())));
            } else {
                LogLog.error("Either File or DatePattern options are not set for appender [" + this.name + "].");
            }
        }
    
        void printPeriodicity(int type) {
            switch (type) {
            case 0:
                LogLog.debug("Appender [" + this.name + "] to be rolled every minute.");
                break;
            case 1:
                LogLog.debug("Appender [" + this.name + "] to be rolled on top of every hour.");
    
                break;
            case 2:
                LogLog.debug("Appender [" + this.name + "] to be rolled at midday and midnight.");
    
                break;
            case 3:
                LogLog.debug("Appender [" + this.name + "] to be rolled at midnight.");
    
                break;
            case 4:
                LogLog.debug("Appender [" + this.name + "] to be rolled at start of week.");
    
                break;
            case 5:
                LogLog.debug("Appender [" + this.name + "] to be rolled at start of every month.");
    
                break;
            default:
                LogLog.warn("Unknown periodicity for appender [" + this.name + "].");
            }
        }
    
        int computeCheckPeriod() {
            RollingCalendar rollingCalendar = new RollingCalendar(gmtTimeZone,Locale.getDefault());
    
            Date epoch = new Date(0L);
            if (this.datePattern != null) {
                for (int i = 0; i <= 5; i++) {
                    SimpleDateFormat simpleDateFormat = new SimpleDateFormat(this.datePattern);
                    simpleDateFormat.setTimeZone(gmtTimeZone);
                    String r0 = simpleDateFormat.format(epoch);
                    rollingCalendar.setType(i);
                    Date next = new Date(rollingCalendar.getNextCheckMillis(epoch));
                    String r1 = simpleDateFormat.format(next);
    
                    if ((r0 != null) && (r1 != null) && (!r0.equals(r1))) {
                        return i;
                    }
                }
            }
            return -1;
        }
    
        public void rollOver() {
            if (this.datePattern == null) {
                this.errorHandler.error("Missing DatePattern option in rollOver().");
                return;
            }
            // System.out.println("rollOver() 1111");
            String datedFilename = this.fileName + this.sdf.format(this.now);
    
            if (this.scheduledFilename.equals(datedFilename)) {
                return;
            }
    
            closeFile();
    
            File target = new File(this.scheduledFilename);
            if (target.exists()) {
                target.delete();
            }
            File file = new File(this.fileName);
    
            // 重命名文件。
            /* * boolean result = file.renameTo(target); if (result) * LogLog.debug(this.fileName + " -> " + this.scheduledFilename); else { * LogLog.error("Failed to rename [" + this.fileName + "] to [" + * this.scheduledFilename + "]."); } */
            // 新建文件
            try {
                SimpleDateFormat formats = new SimpleDateFormat("'" + this.val + "'yyyy-MM-dd-HH'.log'");
                this.fileName = formats.format(new Date());
                setFile(this.fileName, true, this.bufferedIO, this.bufferSize);
            } catch (IOException e) {
                this.errorHandler.error("setFile(" + this.fileName + ", true) call failed.");
            }
            this.scheduledFilename = datedFilename;
        }
    
        protected void subAppend(LoggingEvent event) {
            long n = System.currentTimeMillis();
            if (n >= this.nextCheck) {
                this.now.setTime(n);
                this.nextCheck = this.rc.getNextCheckMillis(this.now);
                rollOver();
            }
            super.subAppend(event);
        }
    }
    
    class RollingCalendar extends GregorianCalendar {
        private static final long serialVersionUID = -3560331770601814177L;
        int type = -1;
    
        RollingCalendar() {
        }
    
        RollingCalendar(TimeZone tz, Locale locale) {
            super(tz, locale);
        }
    
        void setType(int type) {
            this.type = type;
        }
    
        public long getNextCheckMillis(Date now) {
            return getNextCheckDate(now).getTime();
        }
    
        public Date getNextCheckDate(Date now) {
            setTime(now);
    
            switch (this.type) {
            case 0:
                set(13, 0);
                set(14, 0);
                add(12, 1);
                break;
            case 1:
                set(12, 0);
                set(13, 0);
                set(14, 0);
                add(11, 1);
                break;
            case 2:
                set(12, 0);
                set(13, 0);
                set(14, 0);
                int hour = get(11);
                if (hour < 12) {
                    set(11, 12);
                } else {
                    set(11, 0);
                    add(5, 1);
                }
                break;
            case 3:
                set(11, 0);
                set(12, 0);
                set(13, 0);
                set(14, 0);
                add(5, 1);
                break;
            case 4:
                set(7, getFirstDayOfWeek());
                set(11, 0);
                set(12, 0);
                set(13, 0);
                set(14, 0);
                add(3, 1);
                break;
            case 5:
                set(5, 1);
                set(11, 0);
                set(12, 0);
                set(13, 0);
                set(14, 0);
                add(2, 1);
                break;
            default:
                throw new IllegalStateException("Unknown periodicity type.");
            }
            return getTime();
        }
    }

将原来的rename和copy都舍弃掉，然后将文件改成yyyy-MM-dd-HH。  
配置文件引入自定义的日志类：

结果：好了

这里的日志文件格式尽量与配置文件保持一致。不然可能生成的文件名会有问题。但是即使文件名有问题，也不会丢失日志。