Logstash 过滤插件: grok

柔情只为你懂 2022-11-16 04:48 179阅读 0赞

### 文章目录 ###

*  1, Logstash 内置grok模式
 *  2, 自定义grok 模式
 *  3, grok 调试工具： kibana ( 开发工具 --> Grok Debugger)

[https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html][https_www.elastic.co_guide_en_logstash_current_plugins-filters-grok.html]

# 1, Logstash 内置grok模式 #

*  github 模式地址: [https://github.com/logstash-plugins/logstash-patterns-core/tree/master/patterns/legacy][https_github.com_logstash-plugins_logstash-patterns-core_tree_master_patterns_legacy]
 *  安装包内置模式路径：logstash-7.11.2\\vendor\\bundle\\jruby\\2.5.0\\gems\\logstash-patterns-core-4.1.2\\patterns  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2V5ZW9mZWFnbGU_size_16_color_FFFFFF_t_70]

#### 文件名： httpd
    MK@DESKTOP-wang /d/download/elk-stack/logstash-7.11.2/vendor/bundle/jruby/2.5.0/gems/logstash-patterns-core-4.1.2/patterns                                                                                         
    λ grep COMBINEDAPACHELOG *                                                                                                                                                                                         
    httpd:COMBINEDAPACHELOG %{ HTTPD_COMBINEDLOG}    
    
    MK@DESKTOP-wang /d/download/elk-stack/logstash-7.11.2/vendor/bundle/jruby/2.5.0/gems/logstash-patterns-core-4.1.2/patterns                                                                                         
    λ grep ^HTTPD_COMBINEDLOG *                                                                                                                                                                                         
    httpd:HTTPD_COMBINEDLOG %{ HTTPD_COMMONLOG} %{ QS:referrer} %{ QS:agent} 
    
    MK@DESKTOP-wang /d/download/elk-stack/logstash-7.11.2/vendor/bundle/jruby/2.5.0/gems/logstash-patterns-core-4.1.2/patterns                                                                                         
    λ grep ^HTTPD_COMMONLOG *                                                                                                                                                                                          
    httpd:HTTPD_COMMONLOG %{ IPORHOST:clientip} %{ HTTPDUSER:ident} %{ HTTPDUSER:auth} \[%{ HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{ NUMBER:respon
    se} (?:%{ NUMBER:bytes}|-)   
    
    
    
    #### 文件名： grok-patterns
    MK@DESKTOP-wang /d/download/elk-stack/logstash-7.11.2/vendor/bundle/jruby/2.5.0/gems/logstash-patterns-core-4.1.2/patterns                                                                                         
    λ grep ^IPORHOST *                                                                                                                                                                                                 
    grok-patterns:IPORHOST (?:%{ IP}|%{ HOSTNAME})                                                                                                                                                                       
    
    MK@DESKTOP-wang /d/download/elk-stack/logstash-7.11.2/vendor/bundle/jruby/2.5.0/gems/logstash-patterns-core-4.1.2/patterns                                                                                         
    λ grep ^IP *                                                                                                                                                                                                       
    grok-patterns:IPV6 ((([0-9A-Fa-f]{ 1,4}:){ 7}([0-9A-Fa-f]{ 1,4}|:))|(([0-9A-Fa-f]{ 1,4}:){ 6}(:[0-9A-Fa-f]{ 1,4}|((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){ 3})|:))|(([0-9A-Fa-f]{ 1,4}:){ 5}( ((:[0-9A-Fa-f]{ 1,4}){ 1,2})|:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){ 3})|:))|(([0-9A-Fa-f]{ 1,4}:){ 4}(((:[0-9A-Fa-f]{ 1,4}){ 1,3})|((:[0-9A-Fa-f]{ 1,4})?:((25[0-5]|2[0-4]\d|1\d\d|[1-9] ?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){ 3}))|:))|(([0-9A-Fa-f]{ 1,4}:){ 3}(((:[0-9A-Fa-f]{ 1,4}){ 1,4})|((:[0-9A-Fa-f]{ 1,4}){ 0,2}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){ 3}))|:))|(( [0-9A-Fa-f]{ 1,4}:){ 2}(((:[0-9A-Fa-f]{ 1,4}){ 1,5})|((:[0-9A-Fa-f]{ 1,4}){ 0,3}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){ 3}))|:))|(([0-9A-Fa-f]{ 1,4}:){ 1}(((:[0-9A-Fa-f]{ 1,4}){ 1,6})|((:[ 0-9A-Fa-f]{ 1,4}){ 0,4}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){ 3}))|:))|(:(((:[0-9A-Fa-f]{ 1,4}){ 1,7})|((:[0-9A-Fa-f]{ 1,4}){ 0,5}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4] \d|1\d\d|[1-9]?\d)){ 3}))|:)))(%.+)?                                                                                                                                                                                
    grok-patterns:IPV4 (?<![0-9])(?:(?:[0-1]?[0-9]{ 1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{ 1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{ 1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{ 1,2}|2[0-4][0-9]|25[0-5]))(?![0-9])
                                                                                                                                                                                                                       
    grok-patterns:IP (?:%{ IPV6}|%{ IPV4})                                                                                                                                                                               
    grok-patterns:IPORHOST (?:%{ IP}|%{ HOSTNAME})     
    
    MK@DESKTOP-wang /d/download/elk-stack/logstash-7.11.2/vendor/bundle/jruby/2.5.0/gems/logstash-patterns-core-4.1.2/patterns                                                                                         
    λ grep ^HOSTNAME *                                                                                                                                                                                                 
    grok-patterns:HOSTNAME \b(?:[0-9A-Za-z][0-9A-Za-z-]{ 0,62})(?:\.(?:[0-9A-Za-z][0-9A-Za-z-]{ 0,62}))*(\.?|\b)

# 2, 自定义grok 模式 #

[https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html][https_www.elastic.co_guide_en_logstash_current_plugins-filters-grok.html]

### 1, 配置logstash 
    D:\download\elk-stack\logstash-7.11.2\bin                                                                                              
    λ type ..\config\logstash-sample-grok.conf                                                                                                                                                                                                                                                                                                   
    
    input {                                                                                                                                 
       stdin {  }                                                                                                                           
    }                                                                                                                                                                                                                                                                             
                                                                                                                                           
    filter {                                                                                                                                
            grok {                                                                                                                          
                    patterns_dir => ["./patterns"]                                                                                         
                    match => {  
                    	#"message" => "%{SYSLOGBASE} %{POSTFIX_QUEUEID:queue_id}: %{GREEDYDATA:syslog_message}"
                    	"message" => "%{POSTFIX_QUEUEID:queue_id}: %{GREEDYDATA:syslog_message}" 
                    }                    
            }                                                                                                                              
    }                                                                                                                                                                                                                                                                             
                                                                                                                                           
    output {                                                                                                                                
      stdout { }                                                                                                                            
    }                                                                                                                                      
                                                                                                                                           
    D:\download\elk-stack\logstash-7.11.2\bin                                                                                              
    λ ls patterns\                                                                                                                         
    postfix                                                                                                                                
                                                                                                                                           
    D:\download\elk-stack\logstash-7.11.2\bin                                                                                              
    λ type patterns\postfix                                                                                                                
    POSTFIX_QUEUEID [0-9A-F]{ 10,11}    
    
    
    
    ### 2, 启动logstash 
    D:\download\elk-stack\logstash-7.11.2\bin
    λ logstash -f ..\config\logstash-sample-grok.conf --config.reload.automatic      #--config.test_and_exit
    Using JAVA_HOME defined java: D:\software\jdk1.8.0_3
    WARNING, using JAVA_HOME while Logstash distribution comes with a bundled JDK
    Sending Logstash logs to D:/download/elk-stack/logstash-7.11.2/logs which is now configured via log4j2.properties
    [2021-04-06T10:25:42,697][INFO ][logstash.runner          ] Starting Logstash { "logstash.version"=>"7.11.2", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc Java HotSpot(TM) 64-Bit Server VM 25.31-b07 on 1.8.0_31-b13 +indy +jit [mswin32-x86_64]"}
    [2021-04-06T10:25:42,798][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
    [2021-04-06T10:25:44,196][INFO ][org.reflections.Reflections] Reflections took 32 ms to scan 1 urls, producing 23 keys and 47 values
    [2021-04-06T10:25:45,320][INFO ][logstash.javapipeline    ][main] Starting pipeline { :pipeline_id=>"main", "pipeline.workers"=>12, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>1500, "pipeline.sources"=>["D:/download/elk-stack/logstash-7.11.2/config/logstash-sample-grok.conf"], :thread=>"#<Thread:0x2c087e76 run>"}
    [2021-04-06T10:25:45,864][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time { "seconds"=>0.53}
    [2021-04-06T10:25:45,925][INFO ][logstash.javapipeline    ][main] Pipeline started { "pipeline.id"=>"main"}
    The stdin plugin is now waiting for input:
    [2021-04-06T10:25:45,973][INFO ][logstash.agent           ] Pipelines running { :count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
    [2021-04-06T10:25:46,137][INFO ][logstash.agent           ] Successfully started Logstash API endpoint { :port=>9600}
    
    #输入测试数据如下：
    Jan  1 06:25:43 mailserver14 postfix/cleanup[21403]: BEF25A72965: message-id=<20130101142543.5828399CCAF@mailserver14.example.com>
    
    #打印匹配结果如下： 
    {                                                                                                                                                                           
               "message" => " Jan 1 06:25:43 mailserver14 postfix/cleanup[21403]: BEF25A72965: message-id=<20130101142543.5828399CCAF@mailserver14.example.com>\r",           
              "@version" => "1",                                                                                                                                               
            "@timestamp" => 2021-04-06T04:37:39.543Z,                                                                                                                          
                  "host" => "DESKTOP-wang",                                                                                                                                    
              "queue_id" => "BEF25A72965",                                                                                                                                     
        "syslog_message" => "message-id=<20130101142543.5828399CCAF@mailserver14.example.com>\r"                                                                               
    }

# 3, grok 调试工具： kibana ( 开发工具 --> Grok Debugger) #

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2V5ZW9mZWFnbGU_size_16_color_FFFFFF_t_70 1]

[https_www.elastic.co_guide_en_logstash_current_plugins-filters-grok.html]: https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html
[https_github.com_logstash-plugins_logstash-patterns-core_tree_master_patterns_legacy]: https://github.com/logstash-plugins/logstash-patterns-core/tree/master/patterns/legacy
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2V5ZW9mZWFnbGU_size_16_color_FFFFFF_t_70]: /images/20221022/a14c1369210244ed86255231e21cd5a2.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2V5ZW9mZWFnbGU_size_16_color_FFFFFF_t_70 1]: /images/20221022/089a913a844c445880cf11cf90c2b2ac.png