Elasticsearch入门之（五）windows下安装7.9版本Logstash、Filebeat，并收集log4j日志，展示在kibana上

朱雀 2020-10-25 08:21 1020阅读 0赞

一、下载Logstash

官网地址：[https://www.elastic.co/cn/downloads/logstash][https_www.elastic.co_cn_downloads_logstash]

二、下载Filebeat

官网地址：[https://www.elastic.co/cn/downloads/beats/filebeat][https_www.elastic.co_cn_downloads_beats_filebeat]

三、解压后，修改Filebeat的配置文件

打开filebeat-7.9.1-windows-x86\_64\\filebeat.yml

修改paths改成自己日志的地址

- f:\idea_workspace\jee_pub_message\logs\*.log

修改enabled，改true

enabled: true

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzE1MjYwMzE1_size_16_color_FFFFFF_t_70][]

注释elasticsearch

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzE1MjYwMzE1_size_16_color_FFFFFF_t_70 1][]

打开logstash

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzE1MjYwMzE1_size_16_color_FFFFFF_t_70 2][]

四、启动Filebeat

在Filebeat根目录cmd输入以下命令启动

filebeat -e -c filebeat.yml -d "publish"

五、新建Logstash的配置文件

在logstash-7.9.0\\bin下新建文件名为logstash.conf，里面内容如下

input {
      beats {
        port => 5044
      }
    }
    
    filter {
    	grok{
    		match => {
    			"message" => "(?<logdate>\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}) %{IP:client} %{URIPATHPARAM:request}"
    		}
    	}
    }
    
    output {
    	stdout {
          codec => rubydebug
        }
        elasticsearch{
            hosts => ["192.168.10.77:9200"]
            index => "logstash"
        }
    }

注：

（1）logstash这里配置的端口需和filebeat配置文件的端口保持一致

（2）grok可参考官网：[https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html][https_www.elastic.co_guide_en_logstash_current_plugins-filters-grok.html]

六、启动Logstash

在logstash-7.9.0\\bin目录cmd输入以下命令启动

logstash -f logstash.conf

七、log4j是正常的配置，没有做修改，以下是我的配置

maven依赖

log4j.properties配置文件

#log4j.appender.errorrollfile.File=${catalina.base}/log/error/error_office.log
    
    #log4j.appender.inforollfile=org.apache.log4j.RollingFileAppender
    #log4j.appender.errorrollfile.MaxFileSize=10MB
    
    ### 设置###
    log4j.rootLogger = info,stdout,D,E
    file.path=${user.dir}
    
    ### 输出信息到控制抬 ###
    log4j.appender.stdout = org.apache.log4j.ConsoleAppender
    log4j.appender.stdout.Target = System.out
    log4j.appender.stdout.layout = org.apache.log4j.PatternLayout
    log4j.appender.stdout.layout.ConversionPattern = [%-5p] %d{yyyy-MM-dd HH:mm:ss,SSS} %m%n
    
    ### 输出DEBUG 级别以上的日志到=E://logs/error.log ###
    log4j.appender.D = org.apache.log4j.DailyRollingFileAppender
    log4j.appender.D.File = ${file.path}/logs/log.log
    log4j.appender.D.Append = true
    log4j.appender.D.Threshold = DEBUG
    log4j.appender.D.Encoding=UTF-8
    log4j.appender.D.layout = org.apache.log4j.PatternLayout
    log4j.appender.D.layout.ConversionPattern = %-d{yyyy-MM-dd HH:mm:ss}  [ %t:%r ] - [ %p ]  %m%n
    
    ### 输出ERROR 级别以上的日志到=E://logs/error.log ###
    log4j.appender.E = org.apache.log4j.DailyRollingFileAppender
    log4j.appender.E.File =${file.path}/logs/error.log 
    log4j.appender.E.Append = true
    log4j.appender.E.Threshold = ERROR 
    log4j.appender.E.Encoding=UTF-8
    log4j.appender.E.layout = org.apache.log4j.PatternLayout
    log4j.appender.E.layout.ConversionPattern = %-d{yyyy-MM-dd HH:mm:ss}  [ %t:%r ] - [ %p ]  %m%n

LogUtils工具类

package com.enter.net.util;
    
    import org.apache.log4j.LogManager;
    import org.apache.log4j.Logger;
    import org.apache.log4j.PropertyConfigurator;
    
    import java.io.BufferedInputStream;
    import java.io.File;
    import java.io.FileInputStream;
    import java.io.IOException;
    import java.util.Properties;
    
    /**
     * log4j工具类
     */
    public class LogUtils {
        private static Logger errorLogger;
        private static Logger warnLogger;
        private static Logger infoLogger;
        private static Logger debugLogger;
    
        static {
            //初始化log4j
            Properties properties = new Properties();
            try {
                System.out.println(System.getProperty("user.dir"));
                properties.load(new BufferedInputStream(new FileInputStream(System.getProperty("user.dir") + File.separator + "config" + File.separator + "log4j.properties")));
                PropertyConfigurator.configure(properties);
            } catch (IOException e) {
                e.printStackTrace();
            }
            errorLogger = LogManager.getLogger("jee_errorLogger");
            infoLogger = LogManager.getLogger("jee_infoLogger");
            debugLogger = LogManager.getLogger("jee_debugLogger");
            warnLogger = LogManager.getLogger("jee_warnLogger");
        }
    
        public static void info(String className, String methodName, String msg) {
            infoLogger.info("类：" + className + "\r\n" + "方法：" + methodName + "\r\n" + msg);
        }
    
        public static void error(String className, String methodName, String msg) {
            errorLogger.error("类：" + className + "\r\n" + "方法：" + methodName + "\r\n" + msg);
        }
    
        public static void debug(String className, String methodName, String msg) {
            debugLogger.debug("类：" + className + "\r\n" + "方法：" + methodName + "\r\n" + msg);
        }
    
        public static void warn(String className, String methodName, String msg) {
            infoLogger.warn("类：" + className + "\r\n" + "方法：" + methodName + "\r\n" + msg);
        }
    
        public static void info(String msg) {
            infoLogger.info(msg);
        }
    
        public static void debug(String msg) {
            debugLogger.debug(msg);
        }
    
        public static void warn(String msg) {
            warnLogger.warn(msg);
        }
    
        public static void warn(String msg, Exception e) {
            warnLogger.warn(msg, e);
        }
    
        public static void error(String msg) {
            errorLogger.error(msg);
        }
    
        public static void error(String msg, Exception e) {
            errorLogger.error(msg, e);
    
        }
    
        public static void error(String msg, Throwable e) {
            errorLogger.error(msg, e);
    
        }
    
    }

八、在Kibana里创建索引模式

（1）点击连接到您的 Elasticsearch 索引

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzE1MjYwMzE1_size_16_color_FFFFFF_t_70 3][]

（2）点击创建索引模式

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzE1MjYwMzE1_size_16_color_FFFFFF_t_70 4][]

（3）输入索引名称点击下一步

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzE1MjYwMzE1_size_16_color_FFFFFF_t_70 5][]

（4）选择按时间戳筛选

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzE1MjYwMzE1_size_16_color_FFFFFF_t_70 6][]

九、添加一条日志信息

LogUtils.info("现在测试一条logstash信息");

idea控制台输出

![20200904113104233.png][]

十、Kibana展示效果

（1）点击discover

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzE1MjYwMzE1_size_16_color_FFFFFF_t_70 7][]

（2）选择刚创建的索引，显示日志信息

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzE1MjYwMzE1_size_16_color_FFFFFF_t_70 8][]

[https_www.elastic.co_cn_downloads_logstash]: https://www.elastic.co/cn/downloads/logstash
[https_www.elastic.co_cn_downloads_beats_filebeat]: https://www.elastic.co/cn/downloads/beats/filebeat
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzE1MjYwMzE1_size_16_color_FFFFFF_t_70]: /images/1603614069250.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzE1MjYwMzE1_size_16_color_FFFFFF_t_70 1]: /images/1603614052165.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzE1MjYwMzE1_size_16_color_FFFFFF_t_70 2]: /images/1603614038795.png
[https_www.elastic.co_guide_en_logstash_current_plugins-filters-grok.html]: https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzE1MjYwMzE1_size_16_color_FFFFFF_t_70 3]: /images/1603614019414.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzE1MjYwMzE1_size_16_color_FFFFFF_t_70 4]: /images/1603613999836.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzE1MjYwMzE1_size_16_color_FFFFFF_t_70 5]: /images/1603613986586.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzE1MjYwMzE1_size_16_color_FFFFFF_t_70 6]: /images/1603613970932.png
[20200904113104233.png]: /images/1603613956579.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzE1MjYwMzE1_size_16_color_FFFFFF_t_70 7]: /images/1603613943074.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzE1MjYwMzE1_size_16_color_FFFFFF_t_70 8]: /images/1603613922530.png