ELK搭建总结

红太狼 2022-12-01 11:46 209阅读 0赞

最近一周得了闲,写一下做的日志监控,分几部分,今天讲一讲环境的搭建

简结表述就是SpringBoot接入ELK+Filebeat收集系统，Kibana展示日志

Elatisc中文社区:  
https://elasticsearch.cn/explore/

## 一 架构图 ##

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjU1NDM3Mw_size_16_color_FFFFFF_t_70_pic_center]

## 二 环境 ##

1、Windows系统（win10环境）  
2、VMware  
3、Centos7  
4、MobaXterm  
5、Docker  
6、Elasticsearch 7.2.0  
7、Kibana 7.2.0  
8、Logstash 7.2.0  
9、Filebeat 7.2.0  
10、SpringBoot

## 三 注意点 ##

① Elasticsearch可以自定义启动内存

find /var/lib/docker -name jvm.options
    得到启动文件的路径,然后vi ..../config/jvm.options

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjU1NDM3Mw_size_16_color_FFFFFF_t_70_pic_center 1]启动es

docker run -d --name es -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" elasticsearch:7.2.0
    docker inspect es 查看IPAddress,后面配置kib用的着
    curl 172.17.0.XX:9200 查看是否启动成功

②kibana启动(确保es已经启动)

启动的时候可以直接指定链接的es地址,也可以在配置文件指定再重启
    docker run --name kib -e ELASTICSEARCH_URL=http://172.17.0.2:9200 -p 5601:5601 -d kibana:7.2.0

在浏览器通过ip:5601打开kib页面,首次打开如果出现报错"kibana server is not ready yet"(非首次打开出现很可能是分配的ip变了,使用docker inspect 容器ID查看es和logstash的IPAddress是不是变了)  
如下操作:

1.进入docker容器
    docker exec -it kib /bin/bash
    2.修改配置文件
    vi config/kibana.yml
    修改连接Elasticsearch地址为 http://172.17.0.2:9200
    3.ctrl+alt+p退出容器并重启
    docker restart kib

③logstash镜像拉取完毕,先不要启动

1.在宿主主机上创建logstash目录mkdir /mnt/cloud/elk/logstash -p
    2.在里面创建logstash.yml和logstash.conf两个文件
    yml配置如下:
    http.port: 5044
    conf配置如下(多个service使用if隔开):
    input { 
      beats { 
         port => 5045
         codec => json
            }
    }
    filter { 
        if [fields][service] == "order-service" { 
           date { 
                match => [ "requestTime" , "yyyy-MM-dd HH:mm:ss" ]
                target => "@timestamp"
            }
           mutate {   
      	   remove_field => "parent"       
      	   remove_field => "meta"    
      	   remove_field => "trace"
    	   remove_field => "tags"
    	   remove_field => "prospector"
           remove_field => "span"
           remove_field => "fields"
           remove_field => "severity"
           remove_field => "@version"
           remove_field => "exportable"
           remove_field => "input"
           remove_field => "pid"
           remove_field => "thread"
           remove_field => "beat"
           remove_field => "host"
           remove_field => "offset"
           remove_field => "log"       
    	}
      }
      if [fields][service] == "userAdmin-service" { 
           date { 
                match => [ "requestTime" , "yyyy-MM-dd HH:mm:ss" ]
                target => "@timestamp"
            }
           mutate { 
               remove_field => "parent"
               remove_field => "meta"
               remove_field => "trace"
               remove_field => "tags"
               remove_field => "prospector"
               remove_field => "span"
               remove_field => "fields"
               remove_field => "severity"
               remove_field => "@version"
               remove_field => "exportable"
               remove_field => "input"
               remove_field => "pid"
               remove_field => "thread"
               remove_field => "beat"
               remove_field => "host"
               remove_field => "offset"
               remove_field => "log"
            }
    
        }
    
    }
    output { 
            elasticsearch { 
                hosts => ["172.17.0.2:9200"]
                index => "%{[esindex]}_%{+YYYYMM}"
      }
    }

启动:

docker run -p 5044:5044 -p 5045:5045 --name lst -d  -v /mnt/cloud/elk/logstash/logstash.conf:/usr/share/logstash/config/logstash.conf -v /mnt/cloud/elk/logstash/logstash.yml:/usr/share/logstash/config/logstash.yml logstash:7.2.0 -f /usr/share/logstash/config/logstash.conf

`docker logs lst`查看启动日志,有如下两个信息说明成功,缺一不可  
![在这里插入图片描述][20200826175042177.png_pic_center]  
`docker inspect lst` 查看启动的IPAddress,后面配置filebeat用的着

④Filebeat安装,下载地址:https://elasticsearch.cn/download/  
tar.gz在/mnt/cloud/elk完成解压,进入文件夹删除原有的配置文件并新增一个配置文件

cd filebeat-7.2.0-linux-x86_64
    rm -rf filebeat.yml
    vim filebeat.yml

配置如下(有多少模块就配置多少):

filebeat.inputs:
    - type: log
      enabled: true
      paths:
        - /elklogs/order-log/*.json fields: service: order-service - type: log enabled: true paths: - /elklogs/userAdmin-log/*.json fields: service: userAdmin-service - type: log enabled: true paths: - /elklogs/pay-log/*.json fields: service: pay-service filebeat.config.modules: path: /mnt/cloud/elk/filebeat-7.2.0-linux-x86_64/modules.d/*.yml reload.enabled: false output.logstash: hosts: ["172.17.0.4:5045"] processors: - add_host_metadata: ~ - add_cloud_metadata: ~

编写filebeat启动脚本,并授权

vim start.sh

内容如下保存退出:

#!/bin/sh
    nohup ./filebeat -e -c filebeat.yml > filebeat.log &

chmod +x start.sh
    
    ./start.sh

查看启动日志,出现\}\}\}\}\}表示成功

tail -f filebeat.log

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjU1NDM3Mw_size_16_color_FFFFFF_t_70_pic_center 3]  
写在最后:  
logstash是cpu密集型操作,es属于io密集型操作,两个东西尽量不要部署到一台服务器上  
后期如果日志量非常大,可以考虑接入kafka(有的架构引入Redis),在filebeat和logstash中间架一个kafka

代码里面怎么去纪录日志,下次有时间再说

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjU1NDM3Mw_size_16_color_FFFFFF_t_70_pic_center]: /images/20221123/d275f5adca824f85afda3d48e650df96.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjU1NDM3Mw_size_16_color_FFFFFF_t_70_pic_center 1]: /images/20221123/f09bce9592cd4d33a1b25eac5f4406a7.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjU1NDM3Mw_size_16_color_FFFFFF_t_70_pic_center 2]: /images/20221123/c4252261632742ea9f1d9a431a089d1d.png
[20200826175042177.png_pic_center]: /images/20221123/1b5587ff100d48878862bbfefac97893.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3dlaXhpbl80MjU1NDM3Mw_size_16_color_FFFFFF_t_70_pic_center 3]: /images/20221123/9f738f8edb6a42f09c801665770bddf8.png