ELK 7.5.0（九）ELK+Filebeat采集多个源日志（自定义索引名称）

超、凢脫俗 2023-06-19 11:27 8阅读 0赞

## 一、实验环境 ##

主机名        IP
    es        192.168.14.210
    kibana    192.168.14.210
    logstash  192.168.14.211
    Filebeat  192.168.14.213
    nginx     192.168.14.213
    secure    192.168.14.213

## 二、安装部署（内容比较多，已经分多个章节） ##

1、ELK（elasticsearch+logstash+kibana）

具体查看第二、三、四章节

2、Filebeat安装

具体查看第六章节

### 三、Filebeat采集多个类型日志 ###

如果客户端有多个不同类型的日志需要一起采集发送到ES服务器，Filebeat配置文件里面需要加入一个字段用来给Logstash区分识别不同类型日志源。

1、修改Filebeat配置文件（添加新字段type）

[root@filebeat ~]# vi /usr/local/filebeat-7.5.0/filebeat.yml
    filebeat.inputs:
    - type: log
      enabled: true
      tail_files: true
      backoff: "1s"
      paths:
          - /usr/local/nginx/logs/access.json.log        #nginx日志路径
      fields:
        filetype: nginx_access
      fields_under_root: true                            #开启自定义字段
    
    - type: log
      enabled: true
      tail_files: true
      backoff: "1s"
      paths:
          - /var/log/secure                               #系统登录日志路径
      fields:
        filetype: linux_secure
      fields_under_root: true
    
    
    output:
      logstash:
        hosts: ["192.168.14.211:5044"]

2、启动或重启服务

[root@filebeat ~]# pkill filebeat
    [root@filebeat ~]# nohup filebeat -e -c /usr/local/filebeat-7.5.0/filebeat.yml > /tmp/filebeat.log 2>&1 &

## 四、Logstash识别多个日志 ##

1、修改Logstash配置文件，通过type字段进行判断日志源；可以自定义日志索引的名称（经过测试不能用大写字母）

[root@logstash ~]# cat   /usr/local/logstash-7.5.0/config/logstash.conf
    input {
        beats {
          host => '0.0.0.0'
          port => 5044 
      }
    }
    
    filter {
      if [filetype] == "nginx_access" {                    #判断字段名称
        json {
          source => "message"
          remove_field => ["message","@version","path","input","log","agent","ecs","tags"]
        }
      }
    }
    
    output{
      if [filetype] == "nginx_access" {                    #判断字段名称
        elasticsearch {
          hosts => ["http://192.168.14.210:9200"]
          user => "elastic"
          password => "elkpwd"
          index => "nginx_access-%{+YYYY.MM.dd}"        #自定义索引名称
        }
      }
      else if [filetype] == "linux_secure" {                #判断字段名称
        elasticsearch {
          hosts => ["http://192.168.14.210:9200"]
          user => "elastic"
          password => "elkpwd"
          index => "linux_secure-%{+YYYY.MM.dd}"        #自定义索引名称
        }
      }
    }

2、重启服务

[root@logstash ~]# nohup logstash -f /usr/local/logstash-7.5.0/config/logstash.conf > /tmp/logstash.log 2> /tmp/logstash.log &

## 五、kibana建立不同日志索引 ##

1、查看索引（一定要有新的日志产生，才会出现索引）

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv_size_16_color_FFFFFF_t_70][]

2、添加索引

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv_size_16_color_FFFFFF_t_70 1][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv_size_16_color_FFFFFF_t_70 2][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv_size_16_color_FFFFFF_t_70 3][]

3、查看linux\_secure日志信息

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv_size_16_color_FFFFFF_t_70 4][]

4、查看nginx\_access日志信息

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv_size_16_color_FFFFFF_t_70 5][]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20191207211144776.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20191207211934129.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/20191207212105371.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv_size_16_color_FFFFFF_t_70 3]: https://img-blog.csdnimg.cn/2019120721200833.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv_size_16_color_FFFFFF_t_70 4]: https://img-blog.csdnimg.cn/20191207212346779.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv_size_16_color_FFFFFF_t_70 5]: https://img-blog.csdnimg.cn/2019120721385784.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3RsYWRhZ2lv,size_16,color_FFFFFF,t_70