基于logstash实现日志文件同步elasticsearch

古城微笑少年丶 2022-12-25 09:00 120阅读 0赞

更多编程教程请到：[菜鸟教程][Link 1] https://www.piaodoo.com/

友情链接：

[高州阳光论坛][Link 2]https://www.hnthzk.com/

[人人影视][Link 3]http://www.op-kg.com/

引言：

之前博文介绍过了mysql/oracle与ES之间的同步机制。而logstash最初始的日志同步功能还没有介绍。本文就logstash同步日志到ES做下详细解读。

**1、目的：**

将本地磁盘存储的日志文件同步（全量同步、实时增量同步）到ES中。

![Image 1][]

**2、源文件：**

[root@5b9dbaaa148a test_log]# ll
    -rwxrwxrwx 1 root root 170 Jul 5 08:02 logmachine.sh
    -rw-r--r-- 1 root root 66 Jul 5 08:25 MProbe01.log
    -rw-r--r-- 1 root root 74 Jul 5 08:28 MProbe02.log

**3、增量实时同步脚本：**

[root@5b9dbaaa148a test_log]# cat logmachine.sh
    #!/bin/bash
    icnt=0;
    while (true)
    do
     echo "[debug][20160703-15:00]"$icnt >> MProbe01.log
     echo "[ERROR][20160704-17:00]"$icnt >> MProbe02.log
     icnt=$((icnt+1));
    done

**4、logstash配置文件：**

[root@5b9dbaaa148a logstash_jdbc_test]# cat log_test.conf
    input {
     file {
     path=> [ "/usr/local/logstash/bin/test_log/MProbe01.log",
    "/usr/local/logstash/bin/test_log/MProbe02.log" ]
     #codec=>multiline {
     # pattern => "^\s"
     # what=>"previous"
     #}
     type=>"probe_log" #类型名称
     # tags=>["XX.XX.XX.XX"]
     }
    }

\#\#\#过滤  
\#filter\{

# grok \{ #

# match => \[“message”,“mailmonitor”\] #

# add\_tag => \[mailmonitor\] #

# \} #

# grok \{ #

# match => \[ “message”, “smsmonitor” \] #

# add\_tag => \[smsmonitor\] #

# \} #

# … #

\#\}

\#\#\#output to es  
output \{  
elasticsearch \{  
hosts => “10.8.5.101:9200”  
index => “mprobe\_index” \#索引名称  
\#template\_name => “mprobelog”  
\#document\_id => “%\{id\}”  
\}  
stdout \{ codec => json\_lines \}  
\}

**5、同步测试：**

> \[root@5b9dbaaa148a bin\]\# ./logstash -f ./logstash\_jdbc\_test/log\_test.conf  
> Settings: Default pipeline workers: 24  
> Pipeline main started  
> \{"message":"\[DEbug\]\[20160305-15:35\]testing02","@version":"1","@timestamp":"2016-07-05T07:26:08.043Z","path":"/usr/local/logstash/bin/test\_log/MProbe01.log","host":"5b9dbaaa148a"  
>

**6、结果验证**

（1）日志记录：

> \[root@5b9dbaaa148a test\_log\]\# tail -f MProbe01.log  
> \[DEbug\]\[20160305-15:35\]testing02  
> \[DEbug\]\[20160305-15:35\]testing01  
> ^C  
> \[root@5b9dbaaa148a test\_log\]\# tail -f MProbe02.log  
> \[DEbug\]\[20160305-15:35\]testing02\_001  
> \[DEbug\]\[20160305-15:35\]testing02\_003  
>

（2）ES记录

![Image 1][]

以上就是本文的全部内容，希望对大家的学习有所帮助，也希望大家多多支持菜鸟教程www.piaodoo.com。

[Link 1]: https://www.piaodoo.com/
[Link 2]: https://www.hnthzk.com/
[Link 3]: http://www.op-kg.com/
[Image 1]: