elasticsearch logstash filter使用测试

浅浅的花香味﹌ 2023-06-29 10:53 7阅读 0赞

--------------------

### elasticsearch logstash filter使用测试 ###

## ##

### \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* ###

**grok 过滤器**

**语法：%\{pattern\_name：field\_name\}**

pattern\_name：正则表达式，匹配message中的文本

field\_name：匹配文本的name，value为匹配的文本内容

**\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\***

**常用内置表达式：[官网链接][Link 1]**

NUMBER、(?:%\{BASE10NUM\})：10进制整数字符串

BASE16NUM ：16进制整数字符串

WORD、\\b\\w+\\b：匹配字符串

SPACE、\\s\*：空格

DATA、.\*?：任意多个字符

GREEDYDATA、.\*：任意多个字符

IP：匹配ip4或者ip6

HOSTNAME：匹配主机名，如www.baidu.com

TIMESTAMP\_ISO8601：匹配时间戳

LOGLEVEL ：日志输出级别，如info、warn等

**\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\***

**常用设置**

**pattern\_definitions：设置自定义正则表达式，hash形式**

filter {
          grok {
            pattern_definitions => { 
              "pattern_name" => "pattern_value",
              "pattern_name2" => "pattern_value2"
            }
          }
        }

**match：解析message，提取内容存储，hash形式**

filter {
          grok {
            match => { 
              "message" => "%{pattern_name:field_name}"
            }
          }
        }

**id：设置过滤器的id，string形式**

filter {
          grok {
            id => "ABC"
          }
        }

**add\_field：添加字段，hash形式**

filter {
          grok {
            pattern_definitions => { 
              add_field => {
              "foo_%{somefield}" => "Hello world, from %{host}"
              "new_field" => "new_static_value"
              }
            }
          }
        }

**remove\_field：删除字段，array形式**

filter {
          grok {
            remove_field => [ "foo_%{somefield}" ]
          }
        }

**add\_tag：添加tag，array形式**

filter {
          grok {
            add_tag => [ "foo_%{somefield}", "tag_1"]
          }
        }

**remove\_tag：删除tag**

filter {
          grok {
            remove_field => [ "foo_%{somefield}", "my_extraneous_field" ]
          }
        }

## ##

### \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* ###

**示例**

**文本：**

hello world www.baidu.com

**管道配置**

input {
      file {
        path => "/usr/share/logstash/logs/hello2.log"
        start_position => "beginning"
      }
    }
    
    filter {
      grok {
        id => "grok"
    
        pattern_definitions => {
          "p1" => "\S*\s*\S*"
        }
    
        match => {
          "message" => "%{p1:word}\s*%{HOSTNAME:ip}"
        }
    
        add_field => {
          "field" => "%{word}"
          "field2" => "hello2"
        }
    
        add_tag => [
          "tag","tag2"
        ]
    
        remove_field => ["path"]
      }
    
      geoip {
        source => "ip"
      }
    }
    
    output {
      stdout { codec => rubydebug }
    }

**控制台输出**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzMTYyNQ_size_16_color_FFFFFF_t_70][]

### \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* ###

**geoip 过滤器：解析ip地址、hostname****相关信息**

**文本**

www.baidu.com
    www.jd.com

**logstash管道配置**

input {
      file {
        path => "/usr/share/logstash/logs/hello3.log"
        start_position => "beginning"
      }
    }
    
    filter {
      grok {
        match => {
          "message" => "%{HOSTNAME:ip}"
        }
      }
    
      geoip {
        source => "ip"
      }
    }
    
    output {
      stdout { codec => rubydebug }
    }

**控制台输出**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzMTYyNQ_size_16_color_FFFFFF_t_70 1][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzMTYyNQ_size_16_color_FFFFFF_t_70 2][]

[Link 1]: https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/grok-patterns
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzMTYyNQ_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20200113173836906.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzMTYyNQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzMTYyNQ_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/2020011317481742.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzMTYyNQ==,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzMTYyNQ_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/2020011317492321.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MzkzMTYyNQ==,size_16,color_FFFFFF,t_70