ELK+Kafka日志集成环境搭建

忘是亡心i 2023-06-16 13:54 14阅读 0赞

# **ELK是什么** #

**![image.png][]**

*  **ElasticSearch简称ES**，它是一个实时的分布式搜索和分析引擎，它可以用于全文搜索，结构化搜索以及分析。它是一个建立在全文搜索引擎 Apache Lucene 基础上的搜索引擎，使用 Java 语言编写。
 *  **Logstash**是一个具有实时传输能力的数据收集引擎，用来进行数据收集（如：读取文本文件）、解析，并将数据发送给ES。
 *  **Kibana**为 Elasticsearch 提供了分析和可视化的 Web 平台。它可以在 Elasticsearch 的索引中查找，交互数据，并生成各种维度表格、图形。

# **ELK的用途** #

**日志分析并不仅仅包括系统产生的错误日志，异常，也包括业务逻辑，或者任何文本类的分析。而基于日志的分析，能够在其上产生非常多的解决方案，譬如：**

1.  **问题排查**。日志分析技术是问题排查的基石。基于日志做问题排查，还有一个很帅的技术，叫全链路追踪，比如SkyWalking、Cat、Zipkin，也算是日志分析技术里的一种。
2.  **监控和预警**。 日志，监控，预警是相辅相成的。
3.  **关联事件**。多个数据源产生的日志进行联动分析，通过某种分析算法，就能够解决生活中各个问题。
4.  **数据分析**。可以根据日志进行某些数据的分析。

# **架构图** #

**![image.png][image.png 1]**

**搭建Elasticsearch**

[https://blog.csdn.net/qq\_37598011/article/details/103137571][https_blog.csdn.net_qq_37598011_article_details_103137571]

[https://blog.csdn.net/qq\_37598011/article/details/103140376][https_blog.csdn.net_qq_37598011_article_details_103140376]

**我这里用的版本还是6.7**

# 安装NPM #

wget https://npm.taobao.org/mirrors/node/v4.4.7/node-v4.4.7-linux-x64.tar.gz
    tar -zxvf node-v4.4.7-linux-x64.tar.gz
    rm -rf node-v4.4.7-linux-x64.tar.gz
    ln -s /usr/local/node/node-v4.4.7-linux-x64/bin/npm /usr/local/bin/npm
    ln -s /usr/local/node/node-v4.4.7-linux-x64/bin/node /usr/local/bin/node
    npm -v

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3NTk4MDEx_size_16_color_FFFFFF_t_70][]

# **安装elasticsearch-head** #

git clone git://github.com/mobz/elasticsearch-head.git
    cd elasticsearch-head
    npm install

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3NTk4MDEx_size_16_color_FFFFFF_t_70 1][]

**如果要其他机器访问，则需要修改Gruntfile.js，在options对象属性下增加一个hostname属性，值为"\*"**

vim Gruntfile.js

hostname: '*'

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3NTk4MDEx_size_16_color_FFFFFF_t_70 2][]  
**启动**

npm run start

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3NTk4MDEx_size_16_color_FFFFFF_t_70 3][]

# 安装Logstash #

wget https://artifacts.elastic.co/downloads/logstash/logstash-6.7.0.tar.gz
    tar -zxvf logstash-6.7.0.tar.gz 
    cd logstash-6.7.0/config/

**创建logstash-es.conf：**

vim logstash-es.conf

**配置如下：**

input {
      stdin { }
      beats {
        port => 9900
        ssl => false
      }
    }
    output {
        elasticsearch {
            action => "index"
            hosts => "127.0.0.1:9200"
            index  => "logstash-%{+YYYY-MM}"
        }
        stdout { codec=> rubydebug }
    }

# ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3NTk4MDEx_size_16_color_FFFFFF_t_70 4][] #

**启动**

../bin/logstash -f ./logstash-es.conf

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3NTk4MDEx_size_16_color_FFFFFF_t_70 5][]

# 安装kibana #

wget https://artifacts.elastic.co/downloads/kibana/kibana-6.7.0-linux-x86_64.tar.gz
    cd kibana-6.7.0-linux-x86_64/config/

**修改配置文件**

vim kibana.yml

server.port: 5601
    server.host: "192.168.78.133"
    elasticsearch.hosts: ["http://192.168.78.133:9200"]
    xpack.reporting.encryptionKey: "a_random_string"
    xpack.security.encryptionKey: "something_at_least_32_characters"
    i18n.locale: "zh-CN"

**启动**

../bin/kibana

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3NTk4MDEx_size_16_color_FFFFFF_t_70 6][]

**    OK~**

**    设置Index Pattern，因为我们在logstash-es.conf中设置为logstash-%\{+YYYY-MM\}，所以设置为logstash-\*就可以了。**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3NTk4MDEx_size_16_color_FFFFFF_t_70 7][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3NTk4MDEx_size_16_color_FFFFFF_t_70 8][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3NTk4MDEx_size_16_color_FFFFFF_t_70 9][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3NTk4MDEx_size_16_color_FFFFFF_t_70 10][]

文档：[https://www.elastic.co/guide/index.html][https_www.elastic.co_guide_index.html]

创建Index：[https://www.elastic.co/guide/en/kibana/6.7/tutorial-load-dataset.html][https_www.elastic.co_guide_en_kibana_6.7_tutorial-load-dataset.html]

批量导入数据到Kibana：[https://www.cnblogs.com/hai-ping/p/6068946.html][https_www.cnblogs.com_hai-ping_p_6068946.html]

# 与Kafka集成 #

**Kafka安装：**[https://blog.csdn.net/qq\_37598011/article/details/88980317][https_blog.csdn.net_qq_37598011_article_details_88980317]

**在logstash的config目录下添加logstash\_kafka.conf**

vim logstash_kafka.conf

input {
      kafka{
        bootstrap_servers=>"192.168.78.133:9092"
        topics=>"test"
        codec=>plain
      }
      #stdin { }
    }
    output {
        elasticsearch {
            #action => "index"
            hosts => "127.0.0.1:9200"
            index  => "test-%{+YYYY-MM}"
        }
        #stdout { codec=> rubydebug }
    }

**重启logstash**

../bin/logstash -f  ./logstash_kafka.conf

**在Kibana里创建test索引**

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3NTk4MDEx_size_16_color_FFFFFF_t_70 11][]

**测试通过kafka添加数据**

./kafka-console-producer.sh --broker-list localhost:9092 --topic test

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3NTk4MDEx_size_16_color_FFFFFF_t_70 12][]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3NTk4MDEx_size_16_color_FFFFFF_t_70 13][]

集群方式：[https://cloud.tencent.com/developer/article/1376678][https_cloud.tencent.com_developer_article_1376678]

[image.png]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly9jZG4ubmxhcmsuY29tL3l1cXVlLzAvMjAxOS9wbmcvNDM2OTkzLzE1NzYyMTYwMTIxNTAtMTU5MzE5NGYtYTAyYS00NTRmLTgzYmEtZmIxNjc2YjA3MDBiLnBuZw?x-oss-process=image/format,png
[image.png 1]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly9jZG4ubmxhcmsuY29tL3l1cXVlLzAvMjAxOS9wbmcvNDM2OTkzLzE1NzYyMTU4OTkyMjctOTM0MTFiYTMtZGQyOC00Y2Y3LThkOWEtOTMwNTI3ZWQ0ZTc1LnBuZw?x-oss-process=image/format,png
[https_blog.csdn.net_qq_37598011_article_details_103137571]: https://blog.csdn.net/qq_37598011/article/details/103137571
[https_blog.csdn.net_qq_37598011_article_details_103140376]: https://blog.csdn.net/qq_37598011/article/details/103140376
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3NTk4MDEx_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20191126172309312.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3NTk4MDEx,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3NTk4MDEx_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20191126173911594.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3NTk4MDEx,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3NTk4MDEx_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/20191126174542500.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3NTk4MDEx,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3NTk4MDEx_size_16_color_FFFFFF_t_70 3]: https://img-blog.csdnimg.cn/20191126174616240.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3NTk4MDEx,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3NTk4MDEx_size_16_color_FFFFFF_t_70 4]: https://img-blog.csdnimg.cn/20191126175956420.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3NTk4MDEx,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3NTk4MDEx_size_16_color_FFFFFF_t_70 5]: https://img-blog.csdnimg.cn/20191126181822159.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3NTk4MDEx,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3NTk4MDEx_size_16_color_FFFFFF_t_70 6]: https://img-blog.csdnimg.cn/20191126184438983.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3NTk4MDEx,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3NTk4MDEx_size_16_color_FFFFFF_t_70 7]: https://img-blog.csdnimg.cn/20191127100102369.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3NTk4MDEx,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3NTk4MDEx_size_16_color_FFFFFF_t_70 8]: https://img-blog.csdnimg.cn/20191127100117349.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3NTk4MDEx,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3NTk4MDEx_size_16_color_FFFFFF_t_70 9]: https://img-blog.csdnimg.cn/20191127100156160.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3NTk4MDEx,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3NTk4MDEx_size_16_color_FFFFFF_t_70 10]: https://img-blog.csdnimg.cn/20191127100232836.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3NTk4MDEx,size_16,color_FFFFFF,t_70
[https_www.elastic.co_guide_index.html]: https://www.elastic.co/guide/index.html
[https_www.elastic.co_guide_en_kibana_6.7_tutorial-load-dataset.html]: https://www.elastic.co/guide/en/kibana/6.7/tutorial-load-dataset.html
[https_www.cnblogs.com_hai-ping_p_6068946.html]: https://www.cnblogs.com/hai-ping/p/6068946.html
[https_blog.csdn.net_qq_37598011_article_details_88980317]: https://blog.csdn.net/qq_37598011/article/details/88980317
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3NTk4MDEx_size_16_color_FFFFFF_t_70 11]: https://img-blog.csdnimg.cn/20191127155441843.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3NTk4MDEx,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3NTk4MDEx_size_16_color_FFFFFF_t_70 12]: https://img-blog.csdnimg.cn/20191127155711826.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3NTk4MDEx,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3NTk4MDEx_size_16_color_FFFFFF_t_70 13]: https://img-blog.csdnimg.cn/20191127155556850.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzM3NTk4MDEx,size_16,color_FFFFFF,t_70
[https_cloud.tencent.com_developer_article_1376678]: https://cloud.tencent.com/developer/article/1376678