Linux 大日志文件按时间查询与分割处理

古城微笑少年丶 2023-03-13 02:29 7阅读 0赞

## 按时间查看日志 ##

日志文件有40G：

sed -n '/2020-05-02 15:00:/,/2020-05-02 15:05:/p' catalina.out > 20200502.log

## 大文件分割 ##

### split命令操作 ###

[root@iZ2ze2pmuviyporobybycvZ ~]# split --help
    Usage: split [OPTION]... [INPUT [PREFIX]]
    Output fixed-size pieces of INPUT to PREFIXaa, PREFIXab, ...; default
    size is 1000 lines, and default PREFIX is 'x'.  With no INPUT, or when INPUT
    is -, read standard input.
    
    Mandatory arguments to long options are mandatory for short options too.
      -a, --suffix-length=N   generate suffixes of length N (default 2)
          --additional-suffix=SUFFIX  append an additional SUFFIX to file names
      -b, --bytes=SIZE        put SIZE bytes per output file
      -C, --line-bytes=SIZE   put at most SIZE bytes of lines per output file
      -d, --numeric-suffixes[=FROM]  use numeric suffixes instead of alphabetic;
                                       FROM changes the start value (default 0)
      -e, --elide-empty-files  do not generate empty output files with '-n'
          --filter=COMMAND    write to shell COMMAND; file name is $FILE
      -l, --lines=NUMBER      put NUMBER lines per output file
      -n, --number=CHUNKS     generate CHUNKS output files; see explanation below
      -u, --unbuffered        immediately copy input to output with '-n r/...'
          --verbose           print a diagnostic just before each
                                output file is opened
          --help     display this help and exit
          --version  output version information and exit
    
    SIZE is an integer and optional unit (example: 10M is 10*1024*1024).  Units
    are K, M, G, T, P, E, Z, Y (powers of 1024) or KB, MB, ... (powers of 1000).
    
    CHUNKS may be:
    N       split into N files based on size of input
    K/N     output Kth of N to stdout
    l/N     split into N files without splitting lines
    l/K/N   output Kth of N to stdout without splitting lines
    r/N     like 'l' but use round robin distribution
    r/K/N   likewise but only output Kth of N to stdout
    
    GNU coreutils online help: <http://www.gnu.org/software/coreutils/>
    For complete documentation, run: info coreutils 'split invocation'

### 按文件行数分割 ###

catalina.out 按50000分割一个文件：

split -l 50000 -d --verbose catalina.out base-split-name
    
    #输出
    base-split-name00
    base-split-name01
    base-split-name02
    ......

### 按文件大小分割 ###

除了按照行数切割之外，split还支持通过文件大小进行切割，通过指定-b参数指定文件大小进行切割，文件大小单位支持K, M, G, T, P, E, Z，如下以切割为500M演示文件切割过程。

split -b 500M -d --verbose catalina.out base-split-name
    
    #输出
    base-split-name00
    
    ......

参考文章：

[Linux运维实战篇之大文件切割][Linux]

[CentOS7按时间段截取指定的Tomcat日志][CentOS7_Tomcat]

[Linux]: https://cloud.tencent.com/developer/article/1576576
[CentOS7_Tomcat]: https://www.jb51.net/article/96341.htm