ELK 搭建

待我称王封你为后i 2021-09-30 14:28 421阅读 0赞

一.环境准备  
两台主机:  
主机1 elk-master 192.168.93.14  
主机2 elk-node 192.168.93.19  
  
主机hosts  
192.168.93.14 elk-master 安装elasticsearch和kibana  
192.168.93.19 elk-node 安装elasticsearch和logstash  
  
备注:版本为6.8.0-1  
  
二.部署  
1.安装jdk  
下载jdk8 地址 [https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html][https_www.oracle.com_technetwork_java_javase_downloads_jdk8-downloads-2133151.html]  
jdk-8u211-linux-x64.tar.gz  
  
\#tar xvf jdk-8u211-linux-x64.tar.gz  
\#mv jdk1.8.0\_211 /usr/local/jdk1.8  
\#vim /etc/profile //添加如下  
export JAVA\_HOME=/usr/local/jdk1.8  
export CLASSPATH=.:$JAVA\_HOME/lib:$JRE\_HOME/lib:$CLASSPATH  
export PATH=$JAVA\_HOME/bin:$JRE\_HOME/bin:$PATH  
export JRE\_HOME=$JAVA\_HOME/jre  
  
\#source /etc/profile  
\#java -version  
java version "1.8.0\_211"  
Java(TM) SE Runtime Environment (build 1.8.0\_211-b12)  
Java HotSpot(TM) 64-Bit Server VM (build 25.211-b12, mixed mode)  
  
\#ln -s /usr/local/jdk1.8/bin/java /usr/bin/  
  
  
2.elasticsearch  
1) 安装  
\[root@elk-master ~\]\# rpm --import [https://artifacts.elastic.co/GPG-KEY-elasticsearch][https_artifacts.elastic.co_GPG-KEY-elasticsearch]  
\[root@elk-master ~\]\# vim /etc/yum.repos.d/elasticsearch.repo //添加如下内容  
\[elasticsearch-6.x\]  
name=Elasticsearch repository for 6.x packages  
baseurl=https://artifacts.elastic.co/packages/6.x/yum  
gpgcheck=1  
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch  
enabled=1  
autorefresh=1  
type=rpm-md  
\[root@elk-master ~\]\# yum install elasticsearch -y  
  
2)配置  
elasticsearch配置文件主要有两个,一个是/etc/elastcisearch/elasticsearch.yml 主要是配置节点信息,另一个是/etc/sysconfig/elasticsearch 配置elasticsearch本身的信息.  
\[root@elk-master ~\]\# vim /etc/elasticsearch/elasticsearch.yml  
cluster.name: lt-elk  
node.name: elk-master  
node.master: true  
node.data: false  
network.host: 0.0.0.0  
http.port: 9200  
discovery.zen.ping.unicast.hosts: \["192.168.93.14", "192.168.93.19"\]  
其他保持默认即可.  
  
数据节点node配置如下  
  
cluster.name: lt-elk  
node.name: elk-node  
node.master: false  
node.data: true  
network.host: 0.0.0.0  
http.port: 9200  
discovery.zen.ping.unicast.hosts: \["192.168.93.14", "192.168.93.19"\]  
  
启动服务  
\[root@elk-master ~\]\# systemctl enable elasticsearch.service  
Created symlink from /etc/systemd/system/multi-user.target.wants/elasticsearch.service to /usr/lib/systemd/system/elasticsearch.service.  
\[root@elk-master ~\]\# systemctl start elasticsearch.service  
  
\[root@elk-node ~\]\# systemctl enable elasticsearch.service  
Created symlink from /etc/systemd/system/multi-user.target.wants/elasticsearch.service to /usr/lib/systemd/system/elasticsearch.service.  
\[root@elk-node ~\]\# systemctl start elasticsearch.service  
  
  
查看下端口  
\[root@elk-master ~\]\# netstat -lntp | grep java  
tcp6 0 0 :::9200 :::\* LISTEN 21817/java  
tcp6 0 0 :::9300 :::\* LISTEN 21817/java  
// 9200 数据传输端口, 9300集群通信端口  
  
测试下:  
\[root@elk-master ~\]\# curl "[http://192.168.93.14:9200"][http_192.168.93.14_9200]  
\{  
"name" : "elk-master",  
"cluster\_name" : "lt-elk",  
"cluster\_uuid" : "PMzMag2vQsanNWbhzu3MZA",  
"version" : \{  
"number" : "6.8.0",  
"build\_flavor" : "default",  
"build\_type" : "rpm",  
"build\_hash" : "65b6179",  
"build\_date" : "2019-05-15T20:06:13.172855Z",  
"build\_snapshot" : false,  
"lucene\_version" : "7.7.0",  
"minimum\_wire\_compatibility\_version" : "5.6.0",  
"minimum\_index\_compatibility\_version" : "5.0.0"  
\},  
"tagline" : "You Know, for Search"  
\}  
查看下集群  
\[root@elk-master ~\]\# curl "[http://192.168.93.14:9200/\_cluster/health?pretty"][http_192.168.93.14_9200_cluster_health_pretty]  
\{  
"cluster\_name" : "lt-elk",  
"status" : "green",  
"timed\_out" : false,  
"number\_of\_nodes" : 2,  
"number\_of\_data\_nodes" : 1,  
"active\_primary\_shards" : 0,  
"active\_shards" : 0,  
"relocating\_shards" : 0,  
"initializing\_shards" : 0,  
"unassigned\_shards" : 0,  
"delayed\_unassigned\_shards" : 0,  
"number\_of\_pending\_tasks" : 0,  
"number\_of\_in\_flight\_fetch" : 0,  
"task\_max\_waiting\_in\_queue\_millis" : 0,  
"active\_shards\_percent\_as\_number" : 100.0  
\}  
  
3.kibana  
安装kibana  
\[root@elk-master ~\]\# yum install kibana -y  
配置  
\[root@elk-master ~\]\# vim /etc/kibana/kibana.yml  
server.port: 5601  
server.host: "192.168.93.14"  
elasticsearch.url: "[http://192.168.93.14:9200"][http_192.168.93.14_9200]  
kibana.index: ".kibana"  
logging.dest: /var/log/kibana.log  
启动  
\[root@elk-master ~\]\# systemctl enable kibana.service  
Created symlink from /etc/systemd/system/multi-user.target.wants/kibana.service to /etc/systemd/system/kibana.service.  
\[root@elk-master ~\]\# systemctl start kibana.service  
  
4.安装logstash  
\[root@elk-node ~\]\# yum install logstash -y  
\[root@elk-node ~\]\# vim /etc/logstash/conf.d/syslog.conf  
input\{  
syslog\{  
tyep => "system-log"  
port => 10514  
\}  
\}  
output\{  
stdout\{  
codec => rubydebug  
\}  
\}  
\[root@elk-node logstash\]\# vim /etc/logstash/logstash.yml  
http.host : 192.168.93.19  
\[root@elk-node logstash\]\# systemctl restart logstash  
验证下配置  
\[root@elk-node ~\]\# cd /usr/share/logstash/bin  
\[root@elk-node bin\]\# ./logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/syslog.conf --config.test\_and\_exit  
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties  
\[2019-06-07T11:59:55,496\]\[WARN \]\[logstash.config.source.multilocal\] Ignoring the 'pipelines.yml' file because modules or command line options are specified  
Configuration OK  
\[2019-06-07T12:00:00,480\]\[INFO \]\[logstash.runner \] Using config.test\_and\_exit mode. Config Validation Result: OK. Exiting Logstash  
\[root@elk-node bin\]\#  
// 显示OK,表示验证  
  
配置rsyslog  
\[root@elk-node logstash\]\# vim /etc/rsyslog.conf  
\*.\* @192.168.93.19:10514  
\[root@elk-node logstash\]\# systemctl restart rsyslog  
  
验证配置  
\[root@elk-node bin\]\# cd /usr/share/logstash/bin/  
\[root@elk-node bin\]\# ./logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/syslog.conf  
另找一台机器ssh登录到note上,  
观察输出,类似如下  
\{  
"facility\_label" => "syslogd",  
"message" => "action 'action 8' resumed (module 'builtin:omfwd') \[v8.24.0-34.el7 try [http://www.rsyslog.com/e/2359][http_www.rsyslog.com_e_2359] \]\\n",  
"@timestamp" => 2019-06-07T14:21:18.000Z,  
"@version" => "1",  
"timestamp" => "Jun 7 22:21:18",  
"logsource" => "elk-node",  
"host" => "192.168.93.19",  
"facility" => 5,  
"type" => "system-log",  
"program" => "rsyslogd",  
"priority" => 46,  
"severity\_label" => "Informational",  
"severity" => 6  
\}  
\{  
"facility\_label" => "security/authorization",  
"message" => "New session 3 of user root.\\n",  
"@timestamp" => 2019-06-07T14:21:18.000Z,  
"@version" => "1",  
"timestamp" => "Jun 7 22:21:18",  
"logsource" => "elk-node",  
"host" => "192.168.93.19",  
"facility" => 4,  
"type" => "system-log",  
"program" => "systemd-logind",  
"priority" => 38,  
"severity\_label" => "Informational",  
"severity" => 6  
\}  
\{  
"facility\_label" => "system",  
"message" => "Started Session 3 of user root.\\n",  
"@timestamp" => 2019-06-07T14:21:18.000Z,  
"@version" => "1",  
"timestamp" => "Jun 7 22:21:18",  
"logsource" => "elk-node",  
"host" => "192.168.93.19",  
"facility" => 3,  
"type" => "system-log",  
"program" => "systemd",  
"priority" => 30,  
"severity\_label" => "Informational",  
"severity" => 6  
\}  
\{  
"facility\_label" => "security/authorization",  
"message" => "pam\_unix(sshd:session): session opened for user root by (uid=0)\\n",  
"@timestamp" => 2019-06-07T14:21:18.000Z,  
"@version" => "1",  
"timestamp" => "Jun 7 22:21:18",  
"logsource" => "elk-node",  
"host" => "192.168.93.19",  
"pid" => "12518",  
"facility" => 10,  
"type" => "system-log",  
"program" => "sshd",  
"priority" => 86,  
"severity\_label" => "Informational",  
"severity" => 6  
\}  
\{  
"facility\_label" => "syslogd",  
"message" => "action 'action 8' resumed (module 'builtin:omfwd') \[v8.24.0-34.el7 try [http://www.rsyslog.com/e/2359][http_www.rsyslog.com_e_2359] \]\\n",  
"@timestamp" => 2019-06-07T14:21:18.000Z,  
"@version" => "1",  
"timestamp" => "Jun 7 22:21:18",  
"logsource" => "elk-node",  
"host" => "192.168.93.19",  
"facility" => 5,  
"type" => "system-log",  
"program" => "rsyslogd",  
"priority" => 46,  
"severity\_label" => "Informational",  
"severity" => 6  
\}  
\{  
"facility\_label" => "security/authorization",  
"message" => "Accepted password for root from 192.168.93.14 port 42568 ssh2\\n",  
"@timestamp" => 2019-06-07T14:21:18.000Z,  
"@version" => "1",  
"timestamp" => "Jun 7 22:21:18",  
"logsource" => "elk-node",  
"host" => "192.168.93.19",  
"pid" => "12518",  
"facility" => 10,  
"type" => "system-log",  
"program" => "sshd",  
"priority" => 86,  
"severity\_label" => "Informational",  
"severity" => 6  
\}  
证明配置成功  
修改配置文件,让收集的日志信息输出到master服务器中，而不是当前终端：  
\[root@elk-node bin\]\# vim /etc/logstash/conf.d/syslog.conf  
input\{  
syslog\{  
type => "system-log"  
port => 10514  
\}  
\}  
output\{  
elasticsearch\{  
hosts => \["192.168.93.14:9200"\]  
index => "system-log-%\{+YYYY.MM\}"  
\}  
\}  
  
检查配置文件,并启动  
\[root@elk-node bin\]\# ./logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/syslog.conf --config.test\_and\_exit  
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties  
\[2019-06-07T22:30:17,139\]\[WARN \]\[logstash.config.source.multilocal\] Ignoring the 'pipelines.yml' file because modules or command line options are specified  
Configuration OK  
\[2019-06-07T22:30:21,755\]\[INFO \]\[logstash.runner \] Using config.test\_and\_exit mode. Config Validation Result: OK. Exiting Logstash  
\[root@elk-node bin\]\# systemctl start logstash  
  
这里要修改下权限,不然虽然能成功启动,但监听不到端口  
//chown logstash /var/log/logstash/logstash-plain.log  
//chown -R logstash /var/lib/logstash/  
  
检查下端口,如下,启动成功  
\[root@elk-node bin\]\# netstat -lntp | grep 9600  
tcp6 0 0 192.168.93.19:9600 :::\* LISTEN 12587/java  
\[root@elk-node bin\]\#  
\[root@elk-node bin\]\# netstat -lntp | grep 10514  
tcp6 0 0 :::10514 :::\* LISTEN 12587/java  
  
5.配置kibana索引  
建立索引index  
![1363095-20190618220345354-1865396190.png][]

![1363095-20190618220402996-569136406.png][]  
  
  
完成后,点击Discover  
![1363095-20190618220414920-7184937.png][]  
  
  
  
使用beats采集日志  
\[root@elk-node2 ~\]\# wget [https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.7.0-x86\_64.rpm][https_artifacts.elastic.co_downloads_beats_filebeat_filebeat-6.7.0-x86_64.rpm]  
\[root@elk-node2 ~\]\# rpm -ivh filebeat-6.7.0-x86\_64.rpm  
准备中... \#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\# \[100%\]  
正在升级/安装...  
1:filebeat-6.7.0-1 \#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\# \[100%\]  
  
修改配置文件  
\[root@elk-node2 ~\]\# vim /etc/filebeat/filebeat.yml  
filebeat.inputs:  
\- type: log  
paths:  
\- /var/log/messages  
output.elasticsearch:  
hosts: \["192.168.93.14:9200"\]  
  
启动filebeat  
\[root@elk-node2 ~\]\# systemctl start filebeat  
  
服务端检查  
\[root@elk-master ~\]\# curl "192.168.93.14:9200/\_cat/indices?v"  
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size  
green open nginx-log-2019.06.07 q5t0CEG8SPajfnuaqQrTYw 5 1 7 0 92.2kb 46.1kb  
green open system-log-2019.06 hG2UzU9AQ2SpQS6tat-gNQ 5 1 27 0 347.2kb 173.6kb  
green open .kibana\_task\_manager bkj5j4MmQf6Kf5wb3\_UeIA 1 1 2 0 25.1kb 12.5kb  
green open .kibana\_1 v5BpxX-JTWqJP3OMyvoClQ 1 1 6 0 57.5kb 28.7kb  
green open filebeat-6.8.0-2019.06.07 MjFxU6A1Q6OhKrU0mNGATw 3 1 0 0 80.6kb 460b  
  
看到出现filebeat-6.8.0-2019.06.07 索引文件表示正常.

转载于:https://www.cnblogs.com/luckyleaf/p/11048113.html

[https_www.oracle.com_technetwork_java_javase_downloads_jdk8-downloads-2133151.html]: https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
[https_artifacts.elastic.co_GPG-KEY-elasticsearch]: https://artifacts.elastic.co/GPG-KEY-elasticsearch
[http_192.168.93.14_9200]: http://192.168.93.14:9200
[http_192.168.93.14_9200_cluster_health_pretty]: http://192.168.93.14:9200/_cluster/health?pretty
[http_www.rsyslog.com_e_2359]: http://www.rsyslog.com/e/2359
[1363095-20190618220345354-1865396190.png]: /images/20210724/4377209ad07d49d084bee7a81af936b7.png
[1363095-20190618220402996-569136406.png]: /images/20210724/8ca0bde1508544c1a47b4adf2f385042.png
[1363095-20190618220414920-7184937.png]: /images/20210724/b4e1703cf552423292fadd7bf0336236.png
[https_artifacts.elastic.co_downloads_beats_filebeat_filebeat-6.7.0-x86_64.rpm]: https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.7.0-x86_64.rpm