ELK日志框架（2）：log4net.ElasticSearch+ Kibana实现日志记录和显示

青旅半醒 2021-11-05 11:18 538阅读 0赞

[ELK日志框架（2）：log4net.ElasticSearch+ Kibana实现日志记录和显示][ELK_2_log4net.ElasticSearch_ Kibana]

原文: [ELK日志框架（2）：log4net.ElasticSearch+ Kibana实现日志记录和显示][ELK_2_log4net.ElasticSearch_ Kibana 1]

# 环境说明 #

1. windows server 2012 R2 64位

2.nbsp;log4net.ElasticSearch

3.nbsp;kibana-5.5.0-windows-x86.zip

# 架构说明 #

数据采集有两种方案Beats或Logstash，Logstash支持[很多种数据源][Link 1]，有log4j但没有log4net，本想使用log4net.redis+redis+logstash的方式实现日志收集。无意间发现了[nbsp;log4net.ElasticSearch][nbsp_log4net.ElasticSearch]可以直接写日志到ES，经过测试非常好用、方便。

![903646-20170707132547972-2141169402.png][]

> log4net.ElasticSearch is a module for thenbsp;[log4net][]nbsp;library to log messages to thenbsp;[ElasticSearch][]nbsp;document database. ElasticSearch offers robust full-text searching an analyzation so that errors and messages can be indexed quickly and searched easily.
> 
> ### Features: ###
> 
>  *  Supports .NET 4.0+
>  *  Easy installation and setup vianbsp;[Nuget][]
>  *  Full support for the Elasticsearch \_bulk API for buffered logging

# 安装kibana #

## 1.下载kibana ##

首先下载windows版本的kibana文件，地址:https://artifacts.elastic.co/downloads/kibana/kibana-5.5.0-windows-x86.zip,解压到E:\\elk\\kibana目录

## 2.配置Kibana ##

打开配置文件E:\\elk\\kibana\\config\\kibana.yml，修改server.host和server.host，如下：

server.host: "192.168.5.175"
    
    elasticsearch.url: "http://192.168.5.175:9200"

## 3.运行Kibana ##

命令行下执行E:\\elk\\kibana\\bin\\kibana.bat文件

![903646-20170712185413478-690638380.png][]

nbsp;

# nbsp;使用log4net.ElasticSearch #

## 1.安装log4net.ElasticSearch ##

[log4net.ElasticSearch][nbsp_log4net.ElasticSearch]可以通过Nuget安装，要注意保持和正在使用的log4net版本一致

Install-Package log4net.ElasticSearch -Version 2.3.4

nbsp;[log4net.ES.Example][]是一个测试Demo，修改配置后就能使用。

## 2.配置log4net.config文件 ##

lt;!-- Default style appender. You can change the name but make sure you reference it corectly. Change the type if you're using .NET 4.0 --gt;
        lt;appender name="ElasticSearchAppender" type="log4net.ElasticSearch.ElasticSearchAppender, log4net.ElasticSearch"gt;
          lt;layout type="log4net.Layout.PatternLayout,log4net"gt;
            lt;param name="ConversionPattern" value="%d{ABSOLUTE} %-5p %c{1}:%L - %m%n" /gt;
          lt;/layoutgt;
          
          lt;!-- You can also use scheme, user, password here if you need https and http authentication, see the docs --gt;
          lt;connectionString value="Server=192.168.5.175;Index=log;Port=9200;rolling=true"/gt;
          
          lt;!-- false means do not eject older messages when the buffer is full, try to keep them all. --gt;
          lt;lossy value="false" /gt;
          
          lt;!-- Any buffer gt; 1 will use the Elasticsearch _bulk API. Tune as needed depending on how high volume your app is and how fast you want messages to show up in ES --gt;
          lt;bufferSize value="1" /gt;
          
          lt;!-- When an event comes in that matches or is above the treshold, the buffer will flush. I don't reccommend setting to "ALL" or it will defeat the purpose of having a buffer --gt;
          lt;evaluator type="log4net.Core.LevelEvaluator"gt;
            lt;threshold value="ALL"/gt;
          lt;/evaluatorgt;
        lt;/appendergt;

Index=log：索引的名字
    rolling=true:每天生成一个索引，比如log-2017.07.13
    bufferSize :批量提交到ES,这里为了演示设置成1

lt;!-- Common to all log4net configs. See log4net docs for more details  --gt;
        lt;rootgt;
          lt;level value="ALL"/gt;
          lt;appender-ref ref="ElasticSearchAppender" /gt;
        lt;/rootgt;

nbsp;

## 2.写入日志 ##

运行测试程序，即可写入几条日志到ES中。

--查询索引列表
    http://192.168.5.175:9200/_cat/indices
    返回值：
    yellow open log-2017.07.13 UBT3YG2qQpWkfJKRNw-XCQ 5 1 1 0 16.5kb 16.5kb
    yellow open log-2017.07.07      pMXjGKkiRRSsrBdl5qNlNw 5 1    4 0  62.6kb  62.6kb
    
    --查询索引内容
    http://192.168.5.175:9200/log-2017.07.13/_search?q=*
    返回值：

{
      "took": 1,
      "timed_out": false,
      "_shards": {
        "total": 5,
        "successful": 5,
        "failed": 0
      },
      "hits": {
        "total": 1,
        "max_score": 1,
        "hits": [
          {
            "_index": "log-2017.07.13",
            "_type": "logEvent",
            "_id": "AV05k9nbXp-PycfFP2XN",
            "_score": 1,
            "_source": {
              "timeStamp": "2017-07-13T01:32:53.9948595Z",
              "message": "System Error",
              "messageObject": {},
              "exception": {
                "Type": "System.Exception",
                "Message": "Something terrible happened.",
                "HelpLink": null,
                "Source": null,
                "HResult": -2146233088,
                "StackTrace": null,
                "Data": {},
                "InnerException": null
              },
              "loggerName": "log4net.ES.Example.Program",
              "domain": "log4net.ES.Example.exe",
              "identity": "",
              "level": "ERROR",
              "className": "log4net.ES.Example.Program",
              "fileName": "F:\\soft\\elk\\log4net.ES.Example-master\\log4net.ES.Example\\Program.cs",
              "lineNumber": "18",
              "fullInfo": "log4net.ES.Example.Program.Main(F:\\soft\\elk\\log4net.ES.Example-master\\log4net.ES.Example\\Program.cs:18)",
              "methodName": "Main",
              "fix": "LocationInfo, UserName, Identity, Partial",
              "properties": {
                "log4net:Identity": "",
                "log4net:UserName": "BJ-BL\\Administrator",
                "log4net:HostName": "BJ-BL",
                "@timestamp": "2017-07-13T01:32:53.9948595Z"
              },
              "userName": "BJ-BL\\Administrator",
              "threadName": "1",
              "hostName": "BJ-BL"
            }
          }
        ]
      }
    }

日志字段非常全。。。。很赞

nbsp;

# 配置kibana #

## 1.创建索引模式 ##

打开kibana，http://192.168.5.175:5601。

点击Management=gt;Index Patterns=gt;Create Index Pattern,输入log-\*就可以匹配上log-yyyy.mm.dd的索引了。

*![903646-20170713094914572-1301500065.png][]*

## 2.查询日志 ##

点击Discover，选中log-\*索引，就可以查询日志了。

![903646-20170713095434103-2019225684.png][]

# 总结 #

ELK日志框架已经搭建完成，这里的L变成了Log4net，ES和Kibana的熟练使用还需要时间去积累。Logstash的插件功能真是强大，也写了几个简单的配置文件实现输入输出的转化，如下：

![ContractedBlock.gif][] ![ExpandedBlockStart.gif][]

input { 
      redis {
            host =gt; "192.168.5.175"    
            port =gt; 6379  
            data_type =gt; "list"     
            key =gt; "logstash" 
            codec=gt; json
        }
    
     }
    output {
      elasticsearch { 
      hosts =gt; ["192.168.5.175:9200"] 
      }
      stdout { codec =gt; rubydebug }
    }

logstash-redis2es.conf

![ContractedBlock.gif][] ![ExpandedBlockStart.gif][]

input { stdin { } }
    output {
      elasticsearch { hosts =gt; ["192.168.5.175:9200"] }
      stdout { codec =gt; rubydebug }
    }

logstash-simple.conf

![ContractedBlock.gif][] ![ExpandedBlockStart.gif][]

input { stdin { } }
    output {
      elasticsearch { hosts =gt; ["192.168.5.175:9200"] }
      file{path=gt;[
      "stdin-log.txt"
      ]}
      stdout { codec =gt; rubydebug }
    }

logstash-stdin2file.conf

![ContractedBlock.gif][] ![ExpandedBlockStart.gif][]

input { stdin { } }
    output {
      redis {
            host =gt; "192.168.5.175"    
            port =gt; 6379  
            data_type =gt; "channel"     
            key =gt; "logstash_list_0"  
        }
       
      stdout { codec =gt; rubydebug }
    }

logstash-stdin2redis.conf

nbsp;

用到的参考资料如下:

1.[https://www.digitalocean.com/community/tutorials/how-to-use-kibana-dashboards-and-visualizations\#kibana-dashboard][https_www.digitalocean.com_community_tutorials_how-to-use-kibana-dashboards-and-visualizations_kibana-dashboard]

2.[https://www.elastic.co/guide/en/logstash/current/plugins-inputs-redis.html\#\_description\_30][https_www.elastic.co_guide_en_logstash_current_plugins-inputs-redis.html_description_30]\\

3.[https://www.elastic.co/products/kibana][https_www.elastic.co_products_kibana],需要一行一行仔细看

4.[ELK部署详解][ELK]，CentOS7环境

5.[logstash配合redis实例][logstash_redis]nbsp;

6.[转：ElasticSearch 索引查询使用指南mdash;mdash;详细版][ElasticSearch _mdash_mdash]

nbsp;

posted on 2019-07-25 11:06 [NET未来之路][NET] 阅读( ...) 评论( ...) [编辑][Link 2] 收藏

转载于:https://www.cnblogs.com/lonelyxmas/p/11242945.html

[ELK_2_log4net.ElasticSearch_ Kibana]: https://www.cnblogs.com/lonelyxmas/p/11242945.html
[ELK_2_log4net.ElasticSearch_ Kibana 1]: https://www.cnblogs.com/zeroes/p/elk-log4net-kibana.html
[Link 1]: https://www.elastic.co/guide/en/logstash/current/input-plugins.html
[nbsp_log4net.ElasticSearch]: https://github.com/jptoto/log4net.ElasticSearch
[903646-20170707132547972-2141169402.png]: /images/20211105/66ace86f6b35446fa37c8e41a9e06c1e.png
[log4net]: http://logging.apache.org/log4net/
[ElasticSearch]: http://www.elasticsearch.org/
[Nuget]: https://nuget.org/packages/log4net.ElasticSearch/
[903646-20170712185413478-690638380.png]: /images/20211105/dc60ce5a75e2452587cc19dd09f24d3e.png
[log4net.ES.Example]: https://github.com/jptoto/log4net.ES.Example
[903646-20170713094914572-1301500065.png]: /images/20211105/e78b9b14fdef46b98fa04f88cf90c39b.png
[903646-20170713095434103-2019225684.png]: /images/20211105/5de24cb34706416695e1940b2621ddb0.png
[ContractedBlock.gif]: https://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif
[ExpandedBlockStart.gif]: /images/20211105/6058d77807f7412285264c3d33ee8ea9.png
[https_www.digitalocean.com_community_tutorials_how-to-use-kibana-dashboards-and-visualizations_kibana-dashboard]: https://www.digitalocean.com/community/tutorials/how-to-use-kibana-dashboards-and-visualizations#kibana-dashboard
[https_www.elastic.co_guide_en_logstash_current_plugins-inputs-redis.html_description_30]: https://www.elastic.co/guide/en/logstash/current/plugins-inputs-redis.html#_description_30
[https_www.elastic.co_products_kibana]: https://www.elastic.co/products/kibana
[ELK]: https://segmentfault.com/a/1190000007728789
[logstash_redis]: http://blog.csdn.net/woshiwusonghaha/article/details/52850826
[ElasticSearch _mdash_mdash]: http://www.cnblogs.com/xuxy03/articles/6031449.html
[NET]: https://www.cnblogs.com/lonelyxmas/
[Link 2]: https://i.cnblogs.com/EditPosts.aspx?postid=11242945