搭建elasticsearch、Kibana、logstash、Filebeat日志查询系统

àì夳堔傛蜴生んèń 2022-03-15 12:56 242阅读 0赞

# 一、环境配置 #

操作系统：CentOS Linux release 7.4.1708

JDK：openjdk version "1.8.0\_191"

ES：elasticsearch-6.6.1

kibana：kibana-6.6.1-linux-x86\_64

logstash：logstash-6.6.1

filebeat：filebeat-6.2.4-linux-x86\_64

测试服务器内网IP：172.1.201.55

测试服务器外网IP：172.1.35.241

测试服务器安装：elasticsearch、kibana、logstash

# 二、安装ES #

下载

创建目录
    mkdir /opt/soft
    进入目录
    cd /opt/soft
    下载
    wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.6.1.tar.gz

安装

tar -zxvf elasticsearch-6.6.1.tar.gz
    mv elasticsearch-6.6.1.tar.gz /tmp/
    cd elasticsearch-6.6.1/
    mkdir /tmp/elasticsearch
    mkdir /tmp/elasticsearch/data
    mkdir /tmp/elasticsearch/logs

编辑配置文件

vim /opt/soft/elasticsearch-6.6.1/config/elasticsearch.yml

cluster.name: payment-application
    node.name: node-1
    path.data: /tmp/elasticsearch/data
    path.logs: /tmp/elasticsearch/logs
    #当前机器的私有IP地址
    network.host: 172.1.201.55
    http.port: 9200

sudo vim /etc/sysctl.conf

添加配置 vm.max\_map\_count=655360

编辑完成，执行命令 sysctl -p

vi /etc/security/limits.conf

在文件最后添加

soft nofile 65536
    hard nofile 65536

创建账号

groupadd elsearch
    useradd elsearch -g elsearch -p /opt/soft/elasticsearch-6.6.1
    chown -R elsearch:elsearch /opt/soft/elasticsearch-6.6.1
    chown -R elsearch:elsearch /tmp/elasticsearch/data
    chown -R elsearch:elsearch /tmp/elasticsearch/logs

启动ES

切换账号
    su elsearch
    /opt/soft/elasticsearch-6.6.1/bin/elasticsearch
    后台进程的方式启动ES：
    /opt/soft/elasticsearch-6.6.1/bin/elasticsearch -d

ES视图

[http://][http]172.1.35.241[:9200/\_cat/health?v][http]

查看所有索引

[http://][http 1]172.1.35.241:[9200/\_cat/indices][http 1]

删除索引

curl -XDELETE '[http://][http 2]172.1.35.241[:9200/xproject\_tenant2\_log4j2\_2019.03.01][http 2]'

ES数据定期删除（还没测试过）

#/bin/bash
    #es-index-clear
    #只保留15天内的日志索引
    LAST_DATA=`date -d "-15 days" "+%Y.%m.%d"`
    #删除上个月份所有的索引,
    curl -XDELETE 'http://ip:port/*-'${LAST_DATA}'*'

# 三、安装Kibana #

下载

cd /opt/soft
    wget https://artifacts.elastic.co/downloads/kibana/kibana-6.6.1-linux-x86_64.tar.gz

安装

tar -zxvf kibana-6.6.1-linux-x86_64.tar.gz 
    vim /opt/soft/kibana-6.6.1-linux-x86_64/config/kibana.yml
    修改配置：
    server.port: 5601
    server.host: "172.1.201.55" #当前机器的私有IP地址
    elasticsearch.hosts: ["http://172.1.201.55:9200"] #ES机器的IP
    kibana.index: ".kibana"

后台启动

nohup /opt/soft/kibana-6.6.1-linux-x86_64/bin/kibana >/dev/null 2>&1 &

kibana访问地址：[http://47.1.213.14:5601][http_47.1.213.14_5601]

## 四、安装logstash ##

下载

cd /opt/soft
    wget https://artifacts.elastic.co/downloads/logstash/logstash-6.6.1.tar.gz

安装

tar -zxvf logstash-6.6.1.tar.gz
    vim /opt/soft/logstash-6.6.1/config/log4j_to_es.conf
    配置内容：
    
    input {
     beats {
       port => 5043
      }
    }
    filter {
    }
    output {
       stdout {
            codec => rubydebug
        }
        #测试环境
        if[fields][log_source] == "xproject_test" {
            elasticsearch {
              hosts => ["172.1.35.241:9200"] #ES的外网IP
              index => "xproject_test_%{+YYYY.MM.dd}"
              manage_template => false
            }
        }
        #正式环境
        if[fields][log_source] == "xproject_tenant1" {
            elasticsearch {
              hosts => ["172.1.35.241:9200"] #ES的外网IP
              manage_template => false
              index => "xproject_tenant1_%{+YYYY.MM.dd}"
            }
        }
    }

启动

nohup /opt/soft/logstash-6.6.1/bin/logstash -f /opt/soft/logstash-6.6.1/config/log4j_to_es.conf >/dev/null 2>&1 &

# 五、安装Filebeat #

下载【测试环境】

在日志收集的服务器上安装
    cd /opt/soft
    wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.2.4-linux-x86_64.tar.gz

安装【测试环境】

tar -zvxf filebeat-6.2.4-linux-x86_64.tar.gz
    编辑配置文件：
    vim /opt/soft/filebeat-6.2.4-linux-x86_64/filebeat.yml
    
    max_procs: 2
    queue.mem:
        events: 128
        flush.min_events: 10
        flush.timeout: 10s
    filebeat.prospectors:
    - input_type: log
      enabled: true
      tail_files: true
      paths:
        - /tmp/logs/XProject_all.log
      fields:
      	log_source: xproject_test
        
    output.logstash:
      hosts: ["172.1.35.241:5043"] #logstash的外网IP

启动

nohup /opt/soft/filebeat-6.2.4-linux-x86_64/filebeat -e -c /opt/soft/filebeat-6.2.4-linux-x86_64/filebeat.yml >/dev/null 2>&1 &

[http]: http://47.91.213.14:9200/_cat/health?v
[http 1]: http://47.91.213.14:9200/_cat/indices
[http 2]: http://IP:9200/xproject_tenant2_log4j2_2019.03.01
[http_47.1.213.14_5601]: http://47.91.213.14:5601/