ELK伪集群+kafka伪集群搭建

待我称王封你为后i 2022-11-07 05:49 401阅读 0赞

文章目录

  • 前言
    • 准备
    • 搭建ES集群
    • 搭建Kibana
    • 搭建Logstash
    • 搭建Kafka 集群
    • 搭建Filebeat

前言

搭建一个ELK伪集群+ kafka伪集群，使用Filebeat收集Nginx的日志，所有应用在一台服务器上搭建，虚拟多块网卡。

  1. ElasticSearch：一个基于 JSON 的分布式的搜索和分析引擎，作为 ELK 的核心，它集中存储数据，用来搜索、分析、存储日志。它是分布式的，可以横向扩容，可以自动发现，索引自动分片。本次搭建3台互为主从的ES集群
  2. Logstash ：一个动态数据收集管道，支持以 TCP/UDP/HTTP 多种方式收集数据（也可以接受 Beats 传输来的数据），并对数据做进一步丰富或提取字段处理。用来采集日志，把日志解析为json格式交给ElasticSearch
  3. Kibana ：一个数据可视化组件，将收集的数据进行可视化展示（各种报表、图形化数据），并提供配置、管理 ELK 的界面
  4. Filebeat ：隶属于Beats,轻量级数据收集引擎。基于原先 Logstash-fowarder 的源码改造出来。换句话说：Filebeat就是新版的 Logstash-fowarder，也会是 ELK Stack 在 Agent 的第一选择。

在这里插入图片描述

准备

一台服务器，虚拟4个ip地址
服务器：Centos7
内存：16GB
ip地址①：192.168.230.133
ip地址①：192.168.230.134
ip地址①：192.168.230.135
ip地址①：192.168.230.136

各ip对应安装的应用和端口号：

  1. 192.168.230.133 虚拟网卡
  2. **********************************
  3. 部署应用                     所用端口
  4. es主节点/es数据节点            9200    9300
  5. head                        9100         
  6. zookeeper                    2888    3888
  7. kibana                        5601  
  8. kafka                       9092
  10. 192.168.230.134 虚拟网卡
  11. **********************************   
  12. 部署应用                     所用端口 
  13. es主节点/es数据节点            9200    9300 
  14. head                        9100     
  15. zookeeper                    2888    3888                 
  16. logstash                     9600  
  17. kafka                       9092
  19. 192.168.230.135 虚拟网卡
  20. **********************************
  21. 部署应用                     所用端口 
  22. es主节点/es数据节点            9200    9300      
  23. zookeeper                     2888    3888  
  24. kafka                       9092
  26. 192.168.230.136 虚拟网卡
  27. **********************************
  28. nginx                        80
  29. Filebeat

所需环境和版本信息如下图，已打包 下载链接：https://download.csdn.net/download/qq_26129413/15728637
在这里插入图片描述

1. 关闭防火墙等

  1. 1.关闭防火墙
  2. systemctl stop firewalld && systemctl disable firewalld
  4. 2. 关闭selinux
  5. sed -i 's/=enforcing/=disabled/g' /etc/selinux/config  && setenforce 0
  7. 3. 修改文件描述符数量，不修改可能使es启动失败
  8. # vim /etc/security/limits.conf
  10. * soft nofile 65536
  11. * hard nofile 131072
  12. * soft nproc 2048
  13. * hard nproc 4096
  15. # vim /etc/sysctl.conf
  17. vm.max_map_count=655360
  19. 5. 刷新 sysctl -p
  20. #将所有安装包放到该目录下
  21. mkdir /software && cd /software

2. 安装JDK1.8环境

  1. tar -zvxf jdk-8u211-linux-x64.tar.gz && mv jdk1.8.0_211/ /usr/local/jdk
  3. # vim /etc/profile
  5. JAVA_HOME=/usr/local/jdk
  6. PATH=$PATH:$JAVA_HOME/bin:$JAVA_HOME/jre/bin
  7. CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/jre/lib
  8. export JAVA_HOME PATH CLASSPATH
  10. # source !$
  12. # java -version
  14. # ln -s /usr/local/jdk/bin/java /usr/local/bin/java

3. 安装 node.js

  1. wget https://nodejs.org/dist/v10.15.3/node-v10.15.3-linux-x64.tar.xz
  4. tar -Jxf node-v10.15.3-linux-x64.tar.xz &&  mv node-v10.15.3-linux-x64/ /usr/local/node
  7. # vim /etc/profile
  9. export NODE_HOME=/usr/local/node
  10. export PATH=$NODE_HOME/bin:$PATH
  11. export NODE_PATH=$NODE_HOME/lib/node_modules:$PATH
  13. # source !$
  15. # node -v

4. 安装head
作用，一个简易的ElasticSearch 分片查询web页面，用来测试es功能是否正常。

  1. #wget  https://github.com/mobz/elasticsearch-head/archive/master.zip
  3. # unzip master.zip && mv elasticsearch-head-master/ /usr/local/elasticsearch-head
  5. # cd /usr/local/elasticsearch-head
  7. # npm install -g cnpm --registry=https://registry.npm.taobao.org
  9. # cnpm install -g grunt-cli
  11. # cnpm install -g grunt
  13. # cnpm install grunt-contrib-clean
  15. # cnpm install grunt-contrib-concat
  17. # cnpm install grunt-contrib-watch
  19. # cnpm install grunt-contrib-connect
  21. # cnpm install grunt-contrib-copy
  23. # cnpm install grunt-contrib-jasmine                #若报错就再执行一遍
  26. -------------------------------------------------------------------------------------------------------
  28.  vim /usr/local/elasticsearch-head/Gruntfile.js
  30. #找到下面connect属性，95行左右，新增 hostname: '192.168.230.133',      #第一台设备ip
  32.                 connect: { 
  33.                         server: { 
  34.                                 options: { 
  35.                                         hostname: '192.168.230.133',            #不要忘了后面的逗号
  36.                                         port: 9100,
  37.                                         base: '.',
  38.                                         keepalive: true
  39.                                 }
  40.                         }
  41.                 }
  43. 后台启动
  44.  cd /usr/local/elasticsearch-head
  46.  nohup grunt server &
  48.  eval "cd /usr/local/elasticsearch-head/ ; nohup npm run start >/dev/null 2>&1 & "
  50. -----------------------------------------------------------------------------------------------------
  52. 后台脚本
  54.  vim /usr/bin/elasticsearch-head
  56. #!/bin/bash
  57. #chkconfig: 2345 55 24
  58. #description: elasticsearch-head service manager
  60. data="cd /usr/local/elasticsearch-head/ ; nohup npm run start >/dev/null 2>&1 & "
  61. START() { 
  62.                 eval $data
  63. }
  65. STOP() { 
  66.                 ps -ef | grep grunt | grep -v "grep" | awk '{print $2}' | xargs kill -s 9 >/dev/null
  67. }
  70. case "$1" in
  71.   start)
  72.         START
  73.         ;;
  74.   stop)
  75.         STOP
  76.         ;;
  77.   restart)
  78.         STOP
  79.         sleep 2
  80.         START
  81.         ;;
  82.   *)
  83.         echo "Usage: elasticsearch-head (|start|stop|restart)"
  84.         ;;
  85. esac
  87. chmod +x /usr/bin/elasticsearch-head
  89. elasticsearch-head  start  启动   elasticsearch-head  stop  停止    elasticsearch-head    restart   重启

5. 创建用户

  1. groupadd   elkms
  2. useradd      elkms  -g  elkms
  3. passwd       elkms  
  4. 123456
  6. groupadd  elklsnoe
  7. useradd     elklsnoe -g  elklsnoe
  8. passwd      elklsnoe
  9. 123456
  11. groupadd elklstow
  12. useradd    elklstow  -g  elklstow
  13. passwd     elklstow
  14. 123456

搭建ES集群

  1. 解压es包
  2. tar zxf elasticsearch-6.7.1.tar.gz && mv elasticsearch-6.7.1 /usr/local/elasticsearchms
  3. tar zxf elasticsearch-6.7.1.tar.gz && mv elasticsearch-6.7.1 /usr/local/elasticsearchlsnoe
  4. tar zxf elasticsearch-6.7.1.tar.gz && mv elasticsearch-6.7.1 /usr/local/elasticsearchlstow
  5. tar zxf elasticsearch-6.7.1.tar.gz && mv elasticsearch-6.7.1 /usr/local/elasticsearchbak      ##多备份了一个
  7. 创建数据目录，并给予权限
  8. [root@localhost local]# mkdir /usr/local/elasticsearchms/data
  9. [root@localhost local]# mkdir /usr/local/elasticsearchlsnoe/data
  10. [root@localhost local]# mkdir /usr/local/elasticsearchlstow/data
  11. [root@localhost local]# chown -R elkms:elkms  /usr/local/elasticsearchms
  12. [root@localhost local]# chown -R elklsnoe:elklsnoe  /usr/local/elasticsearchlsnoe
  13. [root@localhost local]# chown -R elklstow:elklstow  /usr/local/elasticsearchlstow
  15. ##########################################################：修改数据 ms   192.168.230.133
  16. vim /usr/local/elasticsearchms/config/elasticsearch.yml
  18. cluster.name: elk               #集群名，同一集群必须相同
  19. node.name: elk-133              #指定节点主机名
  20. node.master: true               #允许成为主节点
  21. node.data: true                 #数据节点
  22. path.data: /usr/local/elasticsearchms/data                #数据存放路径
  23. path.logs: /usr/local/elasticsearchms/logs                #日志路径
  24. bootstrap.memory_lock: false                #关闭锁定内存，设置为true会报错
  25. network.host: 192.168.230.133                #监听ip
  26. http.port: 9200                 #http端口
  27. transport.tcp.port: 9300
  28. discovery.zen.ping.unicast.hosts: ["192.168.230.133", "192.168.230.134", "192.168.230.135"]                #初始主机列表
  29. discovery.zen.minimum_master_nodes: 2               # n/2+1
  30. http.enabled: true                  #使用http协议对外提供服务
  31. http.cors.enabled: true             #允许head插件访问es
  32. http.cors.allow-origin: "*"
  34. #########################################################修改数据 lsnoe   192.168.230.134
  35. vim /usr/local/elasticsearchlsnoe/config/elasticsearch.yml
  37. cluster.name: elk
  38. node.name: elk-134
  39. node.master: true
  40. node.data: true
  41. path.data: /usr/local/elasticsearchlsnoe/data
  42. path.logs: /usr/local/elasticsearchlsnoe/logs
  43. bootstrap.memory_lock: false
  44. network.host: 192.168.230.134
  45. http.port: 9200
  46. transport.tcp.port: 9300
  47. discovery.zen.ping.unicast.hosts: ["192.168.230.133", "192.168.230.134", "192.168.230.135"]
  48. discovery.zen.minimum_master_nodes: 2
  49. http.enabled: true
  50. http.cors.enabled: true
  51. http.cors.allow-origin: "*"
  53. ########################################################修改数据 lstow   192.168.230.135
  54. vim /usr/local/elasticsearchlstow/config/elasticsearch.yml
  56. cluster.name: elk
  57. node.name: elk-135
  58. node.master: true
  59. node.data: true
  60. path.data: /usr/local/elasticsearchlstow/data
  61. path.logs: /usr/local/elasticsearchlstow/logs
  62. bootstrap.memory_lock: false
  63. network.host: 192.168.230.135
  64. http.port: 9200
  65. transport.tcp.port: 9300
  66. discovery.zen.ping.unicast.hosts: ["192.168.230.133", "192.168.230.134", "192.168.230.135"]
  67. discovery.zen.minimum_master_nodes: 2
  68. http.enabled: true
  69. http.cors.enabled: true
  70. http.cors.allow-origin: "*"
  73. ------------------------------------------------------------------------------------------------------
  74. 修改虚拟内存大小
  75. sed -i 's/-Xms4g/-Xms1g/'   /usr/local/elasticsearchms/config/jvm.options
  76. sed -i 's/-Xmx4g/-Xmx1g/'   /usr/local/elasticsearchms/config/jvm.options
  78. sed -i 's/-Xms4g/-Xms1g/'  /usr/local/elasticsearchlsnoe/config/jvm.options
  79. sed -i 's/-Xmx4g/-Xmx1g/'  /usr/local/elasticsearchlsnoe/config/jvm.options
  81. sed -i 's/-Xms4g/-Xms1g/'  /usr/local/elasticsearchlstow/config/jvm.options
  82. sed -i 's/-Xmx4g/-Xmx1g/'  /usr/local/elasticsearchlstow/config/jvm.options
  84. ---------------------------------------------------------------------------------------------------------
  85. 启动
  86. su - elkms -c "/usr/local/elasticsearchms/bin/elasticsearch -d"
  87. su - elklsnoe -c "/usr/local/elasticsearchlsnoe/bin/elasticsearch -d"
  88. su - elklstow -c "/usr/local/elasticsearchlstow/bin/elasticsearch -d"
  90. 设置开机自启：
  91. vim /etc/rc.local
  92. su - elkms -c "/usr/local/elasticsearchms/bin/elasticsearch -d"
  93. su - elklsnoe -c "/usr/local/elasticsearchlsnoe/bin/elasticsearch -d"
  94. su - elklstow -c "/usr/local/elasticsearchlstow/bin/elasticsearch -d"
  95. chmod +x  /etc/rc.local

搭建Kibana

  1. 解压 kibana  
  2. # tar zxf kibana-6.7.1-linux-x86_64.tar.gz && mv kibana-6.7.1-linux-x86_64 /usr/local/kibana
  4. 修改配置 
  5. # vim /usr/local/kibana/config/kibana.yml
  7. server.port: 5601               #监听端口
  8. server.host: "192.168.230.133"              #监听IP
  9. elasticsearch.hosts: ["http://192.168.230.133:9200","http://192.168.230.134:9200","http://192.168.230.135:9200"]                #集群es地址
  10. logging.dest: /usr/local/kibana/logs/kibana.log                 #日志路径
  11. kibana.index: ".kibana"                 #默认索引
  13. # mkdir /usr/local/kibana/logs && touch /usr/local/kibana/logs/kibana.log
  15. ---------------------------------------------------------------------------------------------------------------------
  16. 启动kibana
  17. /usr/local/kibana/bin/kibana &
  18. 自启
  19. vim  /etc/rc.local
  20. /usr/local/kibana/bin/kibana &

搭建Logstash

  1. 解压
  2. # tar zxf logstash-6.7.1.tar.gz && mv logstash-6.7.1/ /usr/local/logstash
  4. # mkdir /usr/local/logstash/conf.d
  6. 修改配置
  7. #cp /usr/local/logstash/config/logstash.yml /usr/local/logstash/config/logstash.yml.bak
  8. # vim /usr/local/logstash/config/logstash.yml
  10. 36
  11. http.port: 9600
  14. -----------------------------------------------------------------------------------------------
  15. logstash 搜集 搜集Nginx日志配置
  17. # vim /etc/nginx/nginx.d/elk.conf                  # 做一个子配置文件 虚拟主机
  19.     log_format main2 '$http_host $remote_addr - $remote_user [$time_local] "$request" '            #定义日志为main2
  20.                       '$status $body_bytes_sent "$http_referer" '
  21.                       '"$http_user_agent" "$upstream_addr" $request_time';
  23. server { 
  24.       listen 81;
  25.       server_name elk.test.com;
  27.       location / { 
  28.           proxy_pass      http://192.168.230.133:5601;                   # 代理一下 kibana
  29.           proxy_set_header Host   $host;
  30.           proxy_set_header X-Real-IP      $remote_addr;
  31.           proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  32.       }
  34.       access_log  /var/log/nginx/elk_access.log main2;                # 指定日志生成文件
  35. }
  40. # vim /usr/local/logstash/conf.d/nginx_access.conf      #编辑 logstash 配置文件 
  42. input { 
  43.   file { 
  44.     path => "/var/log/nginx/elk_access.log"                 #设置为nginx访问日志的路径
  45.     start_position => "beginning"
  46.     type => "nginx"
  47.   }
  48. }
  49. filter { 
  50.     grok { 
  51.         match => {  "message" => "%{IPORHOST:http_host} %{IPORHOST:clientip} - %{USERNAME:remote_user} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:http_verb} %{NOTSPACE:http_request}(?: HTTP/%{NUMBER:http_version})?|%{DATA:raw_http_request})\" %{NUMBER:response} (?:%{NUMBER:bytes_read}|-) %{QS:referrer} %{QS:agent} %{QS:xforwardedfor} %{NUMBER:request_time:float}"}
  52.     }
  53.     geoip { 
  54.         source => "clientip"
  55.     }
  56. }
  57. output { 
  58.     stdout {  codec => rubydebug }
  59.     elasticsearch { 
  60.         hosts => ["192.168.30.129:9200"]                #也可以为集群内其它机器的地址
  61.         index => "nginx-test-%{+YYYY.MM.dd}"     #设置索引的名字、也是导出的名字
  62.   }
  63. }
  65. --------------------------------------------------------------------------------------------
  66. 启动 logstash
  67. nohup /usr/local/logstash/bin/logstash --path.settings /usr/local/logstash/ -f /usr/local/logstash/conf.d/nginx_access.conf &
  68. 自启
  69. vim  /etc/rc.local
  70. nohup /usr/local/logstash/bin/logstash --path.settings /usr/local/logstash/ -f /usr/local/logstash/conf.d/nginx_access.conf &

搭建Kafka 集群

  1. Zookeeper 配置
  3. cp -r   kafka_2.11-2.1.0.bak   kafka1
  4. cp -r   kafka_2.11-2.1.0.bak   kafka2
  5. cp -r   kafka_2.11-2.1.0.bak   kafka3
  7. su - elkms -c "sed -i 's/^[^#]/#&/' /usr/local/kafka1/config/zookeeper.properties"
  8. su - elklsnoe -c "sed -i 's/^[^#]/#&/' /usr/local/kafka2/config/zookeeper.properties"
  9. su - elklstow -c "sed -i 's/^[^#]/#&/' /usr/local/kafka3/config/zookeeper.properties"
  11. vim /usr/local/kafka1/config/zookeeper.properties
  13. dataDir=/usr/local/kafka1/zookeeper1/data
  14. dataLogDir=/usr/local/kafka1/zookeeper1/logs
  15. clientPort=2181
  16. tickTime=2000
  17. initLimit=20
  18. syncLimit=10
  19. server.1=192.168.230.133:2888:3888
  20. server.2=192.168.230.134:2888:3888
  21. server.3=192.168.230.135:2888:3888
  23. vim /usr/local/kafka2/config/zookeeper.properties
  25. dataDir=/usr/local/kafka2/zookeeper2/data
  26. dataLogDir=/usr/local/kafka2/zookeeper2/logs
  27. clientPort=2182
  28. tickTime=2000
  29. initLimit=20
  30. syncLimit=10
  31. server.1=192.168.230.133:2888:3888
  32. server.2=192.168.230.134:2888:3888
  33. server.3=192.168.230.135:2888:3888
  36. vim /usr/local/kafka3/config/zookeeper.properties
  38. dataDir=/usr/local/kafka3/zookeeper3/data
  39. dataLogDir=/usr/local/kafka3/zookeeper3/logs
  40. clientPort=2183
  41. tickTime=2000
  42. initLimit=20
  43. syncLimit=10
  44. server.1=192.168.230.133:2888:3888
  45. server.2=192.168.230.134:2888:3888
  46. server.3=192.168.230.135:2888:3888
  48. mkdir -p  /usr/local/kafka1/zookeeper1/{ data,logs}
  49. mkdir -p  /usr/local/kafka2/zookeeper2/{ data,logs}
  50. mkdir -p  /usr/local/kafka3/zookeeper3/{ data,logs}
  52. echo 1 > /usr/local/kafka1/zookeeper1/data/myid
  53. echo 2 > /usr/local/kafka2/zookeeper2/data/myid
  54. echo 3 > /usr/local/kafka3/zookeeper3/data/myid
  56. chown -R elkms:elkms /usr/local/kafka1
  57. chown -R elklsnoe:elklsnoe /usr/local/kafka2
  58. chown -R elklstow:elklstow /usr/local/kafka3
  60. -------------------------------------------------------------------------
  62. kafka 配置
  64. sed -i 's/^[^#]/#&/' /usr/local/kafka1/config/server.properties
  65. sed -i 's/^[^#]/#&/' /usr/local/kafka2/config/server.properties
  66. sed -i 's/^[^#]/#&/' /usr/local/kafka3/config/server.properties
  68. vim /usr/local/kafka1/config/server.properties
  70. broker.id=1
  71. listeners=PLAINTEXT://192.168.230.133:9092                  
  72. num.network.threads=3
  73. num.io.threads=8
  74. socket.send.buffer.bytes=102400
  75. socket.receive.buffer.bytes=102400
  76. socket.request.max.bytes=104857600
  77. log.dirs=/usr/local/kafka1/zookeeper1/logs
  78. num.partitions=6
  79. num.recovery.threads.per.data.dir=1
  80. offsets.topic.replication.factor=2
  81. transaction.state.log.replication.factor=1
  82. transaction.state.log.min.isr=1
  83. log.retention.hours=168
  84. log.segment.bytes=536870912
  85. log.retention.check.interval.ms=300000
  86. zookeeper.connect=0.0.0.0:2181,0.0.0.0:2182,0.0.0.0:2183   
  87. zookeeper.connection.timeout.ms=6000
  88. group.initial.rebalance.delay.ms=0
  90. vim /usr/local/kafka2/config/server.properties
  92. broker.id=2
  93. listeners=PLAINTEXT://192.168.230.134:9092
  94. num.network.threads=3
  95. num.io.threads=8
  96. socket.send.buffer.bytes=102400
  97. socket.receive.buffer.bytes=102400
  98. socket.request.max.bytes=104857600
  99. log.dirs=/usr/local/kafka2/zookeeper2/logs
  100. num.partitions=6
  101. num.recovery.threads.per.data.dir=1
  102. offsets.topic.replication.factor=2
  103. transaction.state.log.replication.factor=1
  104. transaction.state.log.min.isr=1
  105. log.retention.hours=168
  106. log.segment.bytes=536870912
  107. log.retention.check.interval.ms=300000
  108. zookeeper.connect=0.0.0.0:2181,0.0.0.0:2182,0.0.0.0:2183
  109. zookeeper.connection.timeout.ms=6000
  110. group.initial.rebalance.delay.ms=0
  112. vim /usr/local/kafka3/config/server.properties
  114. broker.id=3
  115. listeners=PLAINTEXT://192.168.230.135:9092
  116. num.network.threads=3
  117. num.io.threads=8
  118. socket.send.buffer.bytes=102400
  119. socket.receive.buffer.bytes=102400
  120. socket.request.max.bytes=104857600
  121. log.dirs=/usr/local/kafka3/zookeeper3/logs
  122. num.partitions=6
  123. num.recovery.threads.per.data.dir=1
  124. offsets.topic.replication.factor=2
  125. transaction.state.log.replication.factor=1
  126. transaction.state.log.min.isr=1
  127. log.retention.hours=168
  128. log.segment.bytes=536870912
  129. log.retention.check.interval.ms=300000
  130. zookeeper.connect=0.0.0.0:2181,0.0.0.0:2182,0.0.0.0:2183
  131. zookeeper.connection.timeout.ms=6000
  132. group.initial.rebalance.delay.ms=0
  134. ------------------------------------------------
  136. 启动
  137. zookeeper 
  139. su - elkms -c"nohup /usr/local/kafka1/bin/zookeeper-server-start.sh /usr/local/kafka1/config/zookeeper.properties &"
  140. su - elklsnoe -c "nohup /usr/local/kafka2/bin/zookeeper-server-start.sh /usr/local/kafka2/config/zookeeper.properties &"
  141. su - elklstow -c "nohup /usr/local/kafka3/bin/zookeeper-server-start.sh /usr/local/kafka3/config/zookeeper.properties &"

搭建Filebeat

  1. （1）下载
  4.  wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.5.4-linux-x86_64.tar.gz
  7. （2）解压
  10. tar xzvf filebeat-6.5.4-linux-x86_64.tar.gz -C /usr/local/
  11. cd /usr/local/
  12. mv filebeat-6.5.4-linux-x86_64 filebeat
  13. cd filebeat/
  16. （3）修改配置
  18. 修改 Filebeat 配置，支持收集本地目录日志，并输出日志到 Kafka 集群中
  21. [root@es-3-head-kib filebeat]# mv filebeat.yml filebeat.yml.bak
  22. [root@es-3-head-kib filebeat]# vim filebeat.yml
  23. filebeat.prospectors:
  24. - input_type: log        #指定输入的类型
  25.   paths:
  26.     -  /var/log/nginx/*.log #日志的路径 json.keys_under_root: true json.add_error_key: true json.message_key: log output.kafka: hosts: ["192.168.246.234:9092","192.168.246.231:9092","192.168.246.235:9092"] #kafka服务器 topic: 'nginx' #输出到kafka中的topic json.keys_under_root: true #可以让字段位于根节点，默认为false json.add_error_key: true #将解析错误的消息记录储存在error.message字段中 json.message_key: log #message_key是用来合并多行json日志使用的，如果配置该项还需要配置multiline的设置 Filebeat 6.0 之后一些配置参数变动比较大，比如 document_type 就不支持，需要用 fields 来代替等等。 ##### （4）启动 [root@es-3-head-kib filebeat]# nohup ./filebeat -e -c filebeat.yml & [root@es-3-head-kib filebeat]# tail -f nohup.out 2019-08-04T16:55:54.708+0800 INFO kafka/log.go:53 kafka message: client/metadata found some partitions to be leaderless 2019-08-04T16:55:54.708+0800 INFO kafka/log.go:53 client/metadata retrying after 250ms... (2 attempts remaining) ... 验证kafka是否生成topic [root@es-3-head-kib filebeat]# cd /usr/local/kafka_2.11-2.1.0/ [root@es-3-head-kib kafka_2.11-2.1.0]# bin/kafka-topics.sh --zookeeper 192.168.246.231:2181 --list __consumer_offsets nginx #已经生成topic testtopic

现在我们去编辑logstach连接kafka的输出文件

配置完kafka之后查看

在这里插入图片描述

登录到kibana

在这里插入图片描述
在这里插入图片描述

在这里插入图片描述
在这里插入图片描述
在这里插入图片描述

======================================================================================================================================================================================================================================================================================================
辛苦浏览观看，如果对你有帮助，请顺手点个赞吧 (σﾟ∀ﾟ)σ…:*☆

发表评论

表情：
评论列表 （有 0 条评论，401人围观）

还没有评论，来说两句吧...

相关阅读