Promtail+Loki+Grafana搭建轻量级日志管理平台

Myth丶恋晨 2023-10-03 23:51 12阅读 0赞

1. 简介

现在几乎所有的服务都是分布式部署，在定位环境问题时，往往需要在不同的主机上查看日志，并且对于某些行业来说，客户不能接受通过远程登录服务器查看日志。因此，需要一个统一的日志管理平台，可以集中查看日志，并对日子进行过滤、检索、定期清除等。

2. 技术选型

集成日志采集、存储、展示功能的管理平台有很多成熟的解决方案，例如：`Logstash + Elasticsearch + Kibana`，`FileBeat/Fluentd + Elasticsearch + Kibana`，`Promtail + Loki + Grafana`。  
  如果需求复杂，服务器资源不受限制，推荐使用ELK(`Logstash + Elasticsearch + Kibana`)方案；如果需求仅是将不同服务器上的日志采集上来集中展示，且需要一个轻量级的框架，那使用PLG(`Promtail + Loki + Grafana`)最合适不过了。  
  各组件官网：

*  ELK/EFK：[https://www.elastic.co/cn/elastic-stack/][https_www.elastic.co_cn_elastic-stack]
 *  Fluentd：：[Fluentd | Open Source Data Collector | Unified Logging Layer][Fluentd _ Open Source Data Collector _ Unified Logging Layer]
 *  PLG：[https://grafana.com/logs/][https_grafana.com_logs]

3. 相关博客

[SpringBoot 2.x + Prometheus + Grafana 实现应用监控][SpringBoot 2.x _ Prometheus _ Grafana]

4. 安装

本博客使用的`Promtail`和`Loki`版本为：2.6.1，`Grafana`版本为：8.0.7。安装环境为：Linux x86\_64。

*  下载介质和配置文件

<table> 
 <thead> 
  <tr> 
   <th>组件</th> 
   <th>下载地址</th> 
   <th>配置文件地址</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>Promtail</td> 
   <td><a href="https://github.com/grafana/loki/releases/download/v2.6.1/promtail-linux-arm64.zip" title="https://github.com/grafana/loki/releases/download/v2.6.1/promtail-linux-arm64.zip" rel="nofollow">https://github.com/grafana/loki/releases/download/v2.6.1/promtail-linux-arm64.zip</a></td> 
   <td><a href="https://github.com/grafana/loki/blob/v2.6.1/clients/cmd/promtail/promtail-local-config.yaml" title="loki/promtail-local-config.yaml at v2.6.1 · grafana/loki · GitHub" rel="nofollow">loki/promtail-local-config.yaml at v2.6.1 · grafana/loki · GitHub</a></td> 
  </tr> 
  <tr> 
   <td>Loki</td> 
   <td><a href="https://github.com/grafana/loki/releases/download/v2.6.1/loki-linux-arm64.zip" title="https://github.com/grafana/loki/releases/download/v2.6.1/loki-linux-arm64.zip" rel="nofollow">https://github.com/grafana/loki/releases/download/v2.6.1/loki-linux-arm64.zip</a></td> 
   <td><a href="https://github.com/grafana/loki/blob/v2.6.1/cmd/loki/loki-local-config.yaml" title="loki/loki-local-config.yaml at v2.6.1 · grafana/loki · GitHub" rel="nofollow">loki/loki-local-config.yaml at v2.6.1 · grafana/loki · GitHub</a></td> 
  </tr> 
  <tr> 
   <td>Grafana</td> 
   <td><a href="https://dl.grafana.com/enterprise/release/grafana-enterprise-8.0.7.linux-amd64.tar.gz" title="https://dl.grafana.com/enterprise/release/grafana-enterprise-8.0.7.linux-amd64.tar.gz" rel="nofollow">https://dl.grafana.com/enterprise/release/grafana-enterprise-8.0.7.linux-amd64.tar.gz</a></td> 
   <td></td> 
  </tr> 
 </tbody> 
</table>

*  创建安装目录，并将配置文件和介质上传到对应目录

mkdir -p /usr/local/log-manager/promtail
    mkdir -p /usr/local/log-manager/loki
    mkdir -p /usr/local/log-manager/grafana

*  修改`Promtail`配置文件  
      配置参考官网文档：[https://grafana.com/docs/loki/v2.6.x/clients/promtail/configuration/][https_grafana.com_docs_loki_v2.6.x_clients_promtail_configuration]

server:
      http_listen_port: 9080
      grpc_listen_port: 0
    
    # 保存采集日志的位置信息
    positions:
      filename: /home/caolei/log-manager/promtail/tmp/positions.yaml
    
    # Loki接收日志接口
    clients:
      - url: http://localhost:3100/loki/api/v1/push
    
    scrape_configs:
      - job_name: tomcat
        pipeline_stages:
    	# 合并行配置
        - multiline:
            firstline: ^(\d{4})-(\d{2})-(\d{2}) (\d{2}):(\d{2}):(\d{2})
            max_wait_time: 3s
            max_lines: 500
        - match:
            selector: '{job="tomcat"}'
            stages:
    		# 日志过滤，删除DEBUG级别日志
            - drop:
                expression: ".*DEBUG.*"
        static_configs:
        - targets:
            - localhost
          labels:
            job: tomcat   # 给日志富化的标签
            node: node1   # 给日志富化的标签
    		# 采集的日志路径
            __path__: /usr/local/tomcat/logs/catalina.out
    		# 排除的日志路径
    		__path_exclude__: /usr/local/tomcat/logs/{localhost,catalina}.*.log

*  修改`Loki`配置文件  
      配置参考官网文档：[https://grafana.com/docs/loki/v2.6.x/configuration/][https_grafana.com_docs_loki_v2.6.x_configuration]

# 需要运行的组件，默认all。测试中发现，定期删除日志记录需要额外指定：table-manager
    target: "all,table-manager"
    
    auth_enabled: false
    
    server:
      http_listen_port: 3100
      grpc_listen_port: 9096
    
    # 配置Loki存储数据路径
    common:
      path_prefix: /usr/local/loki/data
      storage:
        filesystem:
          chunks_directory: /usr/local/loki/data/chunks
          rules_directory: /usr/local/loki/data/rules
      replication_factor: 1
      ring:
        instance_addr: 127.0.0.1
        kvstore:
          store: inmemory
    
    schema_config:
      configs:
        - from: 2020-10-24
          store: boltdb-shipper
          object_store: filesystem
          schema: v11
    	  # 每24小时存储一个索引目录
          index:
            prefix: index_
            period: 24h
          chunks:
            prefix: chunk_
            period: 24h
    
    storage_config:
      boltdb_shipper:
        active_index_directory: /usr/local/loki/data/boltdb/boltdb-shipper-active
        cache_location: /usr/local/loki/data/boltdb/boltdb-shipper-cache
        cache_ttl: 24h
        shared_store: filesystem
     
    compactor:
      working_directory: /usr/local/loki/data/boltdb/boltdb-shipper-compactor
      shared_store: filesystem
    
    # 限制配置
    limits_config:
      # 拒绝历史数据开关
      reject_old_samples: true
      # 拒绝历史数据最大时长
      reject_old_samples_max_age: 168h
    
    # 表管理配置
    table_manager:
      # 删除历史数据开关
      retention_deletes_enabled: true
      # 删除历史数据时长，不能大于：limits_config.reject_old_samples_max_age
      # 实际上保留时长：(table_manager.retention_period + schema_config.config.chunk.period) <= 保留时长 < (table_manager.retention_period + schema_config.config.chunk.period + 12h)
      retention_period: 168h
    # alert-manager组件，不涉及
    ruler:
      alertmanager_url: http://localhost:9093

*  对安装目录授权

chmod -R 755 /usr/local/log-manager/*

*  启动`Loki`和`Promtail`一定先启动`Loki`再启动`Promtail`

nohup /usr/local/log-manager/loki/loki-linux-amd64 -config.file=/usr/local/log-manager/loki/loki-local-config.yaml >/dev/null 2>/usr/local/log-manager/loki/logs/loki.log 2>&1 &
    
    nohup /usr/local/log-manager/promtail/promtail-linux-amd64 -config.file=/usr/local/log-manager/promtail/promtail-local-config.yaml > /usr/local/log-manager/promtail/logs/promtail.log 2>&1 &

*  启动`Grafana`

cd /usr/local/log-manager/grafana && tar -zxvf grafana-enterprise-8.0.7.linux-amd64.tar.gz
    
    nohup /usr/local/log-manager/grafana/grafana-8.0.7/bin/grafana-server > /dev/null 2>&1 &

*  停止各个组件如果重启了`Loki`，必须同时重启`Promtail`。如果仅重启`Promtail`，可以不重启`Loki`。

# 停止Promtail
    ps -ef | grep promtail | grep promtail-local-config.yaml | grep -v grep | awk '{print $2}'
    
    kill -9 xxx
    
    # 停止Loki
    ps -ef | grep loki | grep loki-local-config.yaml | grep -v grep | awk '{print $2}'
    
    kill -9 xxx
    
    # 停止Grafana
    ps -ef | grep grafana-server
    
    kill -9 xxx

5. 配置Grafana数据源

*  浏览器访问：[http://192.168.0.100:3000][http_192.168.0.100_3000]
 *  登录：admin/admin
 *  Grafana配置数据源

![87df1c0cfd022921bb581afb5f40247b.jpeg][]

![9864daa3d9d84b49244f33ff4262e686.jpeg][]

![5831f5602122127a8a8bfdd0fec9edd5.jpeg][]

*  根据富化的标签，检索日志

![dbc89a6fc0abec7d349c9ad01a509b74.jpeg][]

*  查询语法使用的是`LogQL`语言，学习文档：[https://grafana.com/docs/loki/latest/logql/][https_grafana.com_docs_loki_latest_logql]

[https_www.elastic.co_cn_elastic-stack]: https://www.elastic.co/cn/elastic-stack/
[Fluentd _ Open Source Data Collector _ Unified Logging Layer]: https://www.fluentd.org/
[https_grafana.com_logs]: https://grafana.com/logs/
[SpringBoot 2.x _ Prometheus _ Grafana]: https://www.cnblogs.com/cao-lei/p/14447509.html
[https_grafana.com_docs_loki_v2.6.x_clients_promtail_configuration]: https://grafana.com/docs/loki/v2.6.x/clients/promtail/configuration/
[https_grafana.com_docs_loki_v2.6.x_configuration]: https://grafana.com/docs/loki/v2.6.x/configuration/
[http_192.168.0.100_3000]: http://192.168.0.100:3000
[87df1c0cfd022921bb581afb5f40247b.jpeg]: https://img-blog.csdnimg.cn/img_convert/87df1c0cfd022921bb581afb5f40247b.jpeg
[9864daa3d9d84b49244f33ff4262e686.jpeg]: https://img-blog.csdnimg.cn/img_convert/9864daa3d9d84b49244f33ff4262e686.jpeg
[5831f5602122127a8a8bfdd0fec9edd5.jpeg]: https://img-blog.csdnimg.cn/img_convert/5831f5602122127a8a8bfdd0fec9edd5.jpeg
[dbc89a6fc0abec7d349c9ad01a509b74.jpeg]: https://img-blog.csdnimg.cn/img_convert/dbc89a6fc0abec7d349c9ad01a509b74.jpeg
[https_grafana.com_docs_loki_latest_logql]: https://grafana.com/docs/loki/latest/logql/