安装部署ELK教程 (Elasticsearch+Kibana+Logstash+Filebeat+Metricbeat) 基于7.9.3版本

墨蓝 2022-12-20 02:15 787阅读 0赞

关于docker本地安装参考另外一篇文章 ： [https://blog.csdn.net/u011665991/article/details/109494752][https_blog.csdn.net_u011665991_article_details_109494752]

**docker部署ELK 一台服务器，filebeat和metricbeat另外一台服务器，存在访问拒绝的问题，此问题暂时没有解决所以采用了 linux安装**

安装使用的是官网免安装版本，解压既可以使用，所有使用版本均为7.9.3，压缩包来源于官网。[es官网][es]

### **PS：** ###

### **1、如果因为切换到普通用户没有访问文件夹或者文件的权限，使用root用户重新赋予权限就行** ###

### **        因为在有些配置文件会产生新的文件夹或者新的路径** ###

### **2、关闭防火墙或者开放对应端口的权限** ###

### **3、其他的一些问题，参考文末解决方案** ###

### **4、有些具体错误，需要查看ELK及beat对应的日志** ###

本文部署架构：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE2NjU5OTE_size_16_color_FFFFFF_t_70][]

### **一**、下载安装包 ： ###

链接：[压缩包下载地址][Link 1]  
提取码：elkb

![20201109173104661.png][]

### **二**、上传至服务器： ###

ELK所在的服务器为 100.175

![20201109175455534.png][]

Beat所在的服务器为 100.130

![20201109175849229.png][]

**解压：**

tar -zxvf elasticsearch-7.9.3-linux-x86_64.tar.gz
    tar -zxvf kibana-7.9.3-linux-x86_64.tar.gz
    tar -zxvf logstash-7.9.3.tar.gz
    tar -zxvf filebeat-7.9.3-linux-x86_64.tar.gz
    tar -zxvf metricbeat-7.9.3-linux-x86_64.tar.gz

### **三**、创建用户组 并赋予权限 ###

**elasticsearch和kibana的启动需要用自定义用户（kibana也可以用root启动 ）**

groupadd elk
    useradd elk -g elk -p elk
    chown -R elk:elk /home/tools/elk

### **四**、**修改配置文件：**  ###

**1、修改elasticsearch的配置文件，只补充修改的地方**

**elasticsearch.yml**

# ---------------------------------- Cluster -----------------------------------
    cluster.name: es_175-1
    # ------------------------------------ Node ------------------------------------
    node.name: es_175-1
    # ----------------------------------- Paths ------------------------------------
    
    path.data: /home/tools/elk/elasticsearch-7.9.3/data
    #
    path.logs: /home/tools/elkelasticsearch-7.9.3/logs
    # --------------------------------- Network -----------------------------------
    network.host: 192.168.100.175
    #
    http.port: 9200
    #
    # --------------------------------- Discovery ----------------------------------
    discovery.seed_hosts: ["192.168.100.175"]
    cluster.initial_master_nodes: ["es_175-1"]
    
    http.cors.enabled: true 
    http.cors.allow-origin: "*"
    http.cors.allow-headers: Authorization,Content-Type
    #xpack.security.enabled: true
    #xpack.security.transport.ssl.enabled: true

** 进入到 bin路径下  切换elk用户 启动**

./elasticsearch -d

**访问浏览器：出现此页面成功**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE2NjU5OTE_size_16_color_FFFFFF_t_70 1][]

**设置账户密码：  关于密码设置及修改详细可以参考此外文章：[elasticSearch 设置用户名密码 && 查询][elasticSearch _ _]**

**修改上述注释掉的，放开注释**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE2NjU5OTE_size_16_color_FFFFFF_t_70 2][]

**重新启动**

**执行设置用户名和密码的命令bin路径下，这里需要为4个用户分别设置密码，elastic, kibana, logstash\_system,beats\_system**

./elasticsearch-setup-passwords interactive

**我设置密码时出现的：**

[es@k8snode2 elasticsearch-7.3.0]$ ./bin/elasticsearch-setup-passwords interactive
    Initiating the setup of passwords for reserved users elastic,apm_system,kibana,logstash_system,beats_system,remote_monitoring_user.
    You will be prompted to enter passwords as the process progresses.
    Please confirm that you would like to continue [y/N]y
     
     
    Enter password for [elastic]: 
    Reenter password for [elastic]: 
    Passwords do not match.
    Try again.
    Enter password for [elastic]: 
    Reenter password for [elastic]: 
    Enter password for [apm_system]: 
    Reenter password for [apm_system]: 
    Enter password for [kibana]: 
    Reenter password for [kibana]: 
    Enter password for [logstash_system]: 
    Reenter password for [logstash_system]: 
    Enter password for [beats_system]: 
    Reenter password for [beats_system]: 
    Enter password for [remote_monitoring_user]: 
    Reenter password for [remote_monitoring_user]: 
    Changed password for user [apm_system]
    Changed password for user [kibana]
    Changed password for user [logstash_system]
    Changed password for user [beats_system]
    Changed password for user [remote_monitoring_user]
    Changed password for user [elastic]

**重新访问 此时页面为：**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE2NjU5OTE_size_16_color_FFFFFF_t_70 3][]

**2、修改kibana的配置文件，只补充修改的地方**

**注意：logging.dest: 默认是输出到控制面板，不方便查看日志，可以自定义目录**

**           新路径需要重新给elk普通用户创建文件夹并赋权限、**

**kibana.yml**

server.port: 5601
    
    server.host: "192.168.100.175"
    
    server.name: "kibana_175"
    
    elasticsearch.hosts: ["http://192.168.100.175:9200"]
    
    kibana.index: ".kibana"
    
    # is proxied through the Kibana server.
    elasticsearch.username: "esname"
    elasticsearch.password: "espasswd"
    
    elasticsearch.requestTimeout: 30000
    
    # 注意：此路径需要重新给elk普通用户创建文件夹并赋权限
    #logging.dest: stdout
    logging.dest: /home/tools/elk/kibana-7.9.3-linux-x86_64/logs/kibana.log
    
    # 修改en为zh-CN 控制面板为中文
    i18n.locale: "zh-CN"

** 进入到 bin路径下  切换elk用户 启动**

**如果直接使用 root也能启动会提示添加参数**\--allow-root**，但是启动以后会提示错误[ error in visualization internal server error kibana][error in visualization internal server error kibana]**

**所以这里也要使用普通用户启动**

./kibana &

**注意：此方式启动，控制台会继续输出日志，此时按 ctrl +c 会中断服务，建议使用 nohup后台启动**

nohup ./kibana &

**访问浏览器：出现此页面成功**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE2NjU5OTE_size_16_color_FFFFFF_t_70 4][]

**3、修改logstash的配置文件，只补充修改的地方**

**logstash.yml：**

#
    pipeline.ordered: auto
    #
    xpack.monitoring.enabled: true
    xpack.monitoring.elasticsearch.username: esname
    xpack.monitoring.elasticsearch.password: espasswd

**logstash-sample.conf**

# Sample Logstash configuration for creating a simple
    # Beats -> Logstash -> Elasticsearch pipeline.
    
    input {
      beats {
        port => 5044
      }
    }
    
    output {
    
      if "financials" in [tags] {
         elasticsearch {
            hosts => ["http://192.168.100.175:9200"]
            index => "financials-%{+YYYY.MM.dd}"
            user => "esname"
            password => "espw"
    
         } 
      }
      if "system130" in [tags] {
           elasticsearch {
              hosts => ["http://192.168.100.175:9200"]
              index => "system130-%{+YYYY.MM.dd}"
              user => "esname"
              password => "espw"
    
           }
        }
    
       elasticsearch { 
         hosts => ["http://192.168.100.175:9200"]
         index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
         user => "elastic"
         password => "elastic"
       }
    }

启动logstash:

nohup bin/logstash -f config/logstash-sample.conf &

查看端口状态 此处用的5044，如果没有lsof命名 使用 yum install 安装既可 ，再如果不支持外网 可以用 Telnet 或者curl测试都行

![20201109184402733.png][]

**4、修改metribeat的配置文件，只补充启用的地方**

metricbeat.yml

metricbeat.config.modules:
    
      path: ${path.config}/modules.d/*.yml
      
      reload.enabled: true
    
    # ======================= Elasticsearch template setting =======================
    
    setup.template.settings:
      index.number_of_shards: 1
      index.codec: best_compression
      #_source.enabled: false
    
    # ================================== General ===================================
    #tags: ["service-X", "web-tier"]
    tags: ["system130"]
    
    # ================================= Dashboards =================================
    
    setup.dashboards.enabled: true
    # =================================== Kibana ===================================
    
    setup.kibana:
      host: "192.168.100.175:5601"
      username: "elastic"
      password: "elastic"
    
    # ------------------------------ Logstash Output -------------------------------
    output.logstash:
      hosts: ["192.168.100.175:5044"]
    
    # ================================= Processors =================================
    processors:
      - add_host_metadata: ~
      - add_cloud_metadata: ~
      - add_docker_metadata: ~
      - add_kubernetes_metadata: ~
    # ================================== Logging ===================================
    logging.level: info

**启动metricbeat ,进入到metricbeat 7.9.3文件夹，没有bin目录**

./metricbeat -e

**此命令会经常莫名退出，建议使用：**

nohup ./metricbeat -e -c metricbeat.yml -d publish &

查看服务：

ps -ef | grep metricbeat

### PS：这个地方有个大坑 连接kibana的时候 必须设置用户名和密码，不然的话kibana控制面板 discover可以看到服务指标，但是dashboard始终看不到数据 ###

**5、修改filebeat的配置文件，只补充修改的地方**

filebeat.yml

# ============================== Filebeat inputs ===============================
    
    filebeat.inputs:
    
    
    - type: log
    
      enabled: true
    
      paths:
        # 要监控的服务日志路径
        - /opt/inancials/logs/*.log
        
      # logstash 输出的时候需要使用
      tags: ["financials"]
    
    # ============================== Filebeat modules ==============================
    
    filebeat.config.modules:
    
      path: ${path.config}/modules.d/*.yml
    
      reload.enabled: true
    # ======================= Elasticsearch template setting =======================
    
    setup.template.settings:
      index.number_of_shards: 1
      index.codec: best_compression
      #_source.enabled: false
    
    # ------------------------------ Logstash Output -------------------------------
    output.logstash:
      # The Logstash hosts
      hosts: ["192.168.100.175:5044"]
    # ================================= Processors =================================
    processors:
      - add_host_metadata:
          when.not.contains.tags: forwarded
      - add_cloud_metadata: ~
      - add_docker_metadata: ~
      - add_kubernetes_metadata: ~
    
    # ================================== Logging ===================================
    
    logging.level: info

**启动filebeat ,进入到flebeat 7.9.3文件夹，没有bin目录**

nohup ./filebeat -e

### 五、使用方法 ###

**1、访问kibana的webURL：http:// kibana服务器IP：5601**

**2、正常情况下，kibana控制面板中会发现metricbeat和filebeat的索引**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE2NjU5OTE_size_16_color_FFFFFF_t_70 5][]

**3，但是也不排除查找不到的情况，需要手动添加下索引 ，所有索引都是此操作**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE2NjU5OTE_size_16_color_FFFFFF_t_70 6][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE2NjU5OTE_size_16_color_FFFFFF_t_70 7][]

**4、回到discovery，会发现新增的索引，右侧为服务器日志显示**

**这里举例用的是metricbeat，filebeat同理，这是filebeat没有控制面板**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE2NjU5OTE_size_16_color_FFFFFF_t_70 8][]

**5、显示130服务的默认指标**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE2NjU5OTE_size_16_color_FFFFFF_t_70 9][]

## ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE2NjU5OTE_size_16_color_FFFFFF_t_70 10][] ##

## 6、ELK整合filebeat和metricbeat监控完成， ##

### 这里创建索引有个问题 针对单个项目或者单台服务器没有必要采用   index => "financials-%\{+YYYY.MM.dd\}" 一天创建一个索引的形式 ###

### 具体的按照实际项目来就可以。 ###

## 遇到的一些问题： ##

### 1、[org.elasticsearch.discovery.MasterNotDiscoveredException异常解决 ][org.elasticsearch.discovery.MasterNotDiscoveredException_] ###

### 2、[filebeat启动报错 more than one namespace configured accessing ‘output’][filebeat_ more than one namespace configured accessing _output] ###

### 3、[kibana启动提示："pid":22160,"message":"Port 5601 is already in use. Another instance of Kibana may be running!"\}][kibana_pid_22160_message_Port 5601 is already in use. Another instance of Kibana may be running] ###

### 4、[elasticsearch root账户启动会出现org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException:can not run elasticsearch as root][elasticsearch root_org.elasticsearch.bootstrap.StartupException_ java.lang.RuntimeException_can not run elasticsearch as root] ###

### 5、[其他的一些问题][Link 2] ###

[https_blog.csdn.net_u011665991_article_details_109494752]: https://blog.csdn.net/u011665991/article/details/109494752
[es]: https://www.elastic.co/cn/
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE2NjU5OTE_size_16_color_FFFFFF_t_70]: /images/20221120/4919fbd0461043d1bc2b3f08381e7f0f.png
[Link 1]: https://pan.baidu.com/s/18dKzG05NOc1HIBvsQoxPlg
[20201109173104661.png]: https://img-blog.csdnimg.cn/20201109173104661.png
[20201109175455534.png]: https://img-blog.csdnimg.cn/20201109175455534.png
[20201109175849229.png]: https://img-blog.csdnimg.cn/20201109175849229.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE2NjU5OTE_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/2020110918162443.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE2NjU5OTE=,size_16,color_FFFFFF,t_70
[elasticSearch _ _]: https://blog.csdn.net/mengo1234/article/details/104989382
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE2NjU5OTE_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/20201109182255447.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE2NjU5OTE=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE2NjU5OTE_size_16_color_FFFFFF_t_70 3]: https://img-blog.csdnimg.cn/20201109182747553.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE2NjU5OTE=,size_16,color_FFFFFF,t_70
[error in visualization internal server error kibana]: https://stackoverflow.com/questions/58028361/error-in-visualization-internal-server-error-kibana
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE2NjU5OTE_size_16_color_FFFFFF_t_70 4]: https://img-blog.csdnimg.cn/20201109183646373.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE2NjU5OTE=,size_16,color_FFFFFF,t_70
[20201109184402733.png]: https://img-blog.csdnimg.cn/20201109184402733.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE2NjU5OTE_size_16_color_FFFFFF_t_70 5]: https://img-blog.csdnimg.cn/20201110101535962.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE2NjU5OTE=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE2NjU5OTE_size_16_color_FFFFFF_t_70 6]: https://img-blog.csdnimg.cn/2020111010260826.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE2NjU5OTE=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE2NjU5OTE_size_16_color_FFFFFF_t_70 7]: https://img-blog.csdnimg.cn/20201110102524391.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE2NjU5OTE=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE2NjU5OTE_size_16_color_FFFFFF_t_70 8]: https://img-blog.csdnimg.cn/20201110102847411.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE2NjU5OTE=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE2NjU5OTE_size_16_color_FFFFFF_t_70 9]: https://img-blog.csdnimg.cn/20201110103037979.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE2NjU5OTE=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE2NjU5OTE_size_16_color_FFFFFF_t_70 10]: https://img-blog.csdnimg.cn/20201110103108266.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTE2NjU5OTE=,size_16,color_FFFFFF,t_70
[org.elasticsearch.discovery.MasterNotDiscoveredException_]: https://blog.csdn.net/guanking19/article/details/94414811
[filebeat_ more than one namespace configured accessing _output]: https://blog.csdn.net/qq_42697109/article/details/108092418
[kibana_pid_22160_message_Port 5601 is already in use. Another instance of Kibana may be running]: https://blog.csdn.net/hitits/article/details/84321016
[elasticsearch root_org.elasticsearch.bootstrap.StartupException_ java.lang.RuntimeException_can not run elasticsearch as root]: https://my.oschina.net/sjyz/blog/3101299
[Link 2]: https://www.cnblogs.com/TJ21/p/12638320.html