利用td-agent(fluentd)和elasticsearch搭建kibana可视化日志分析平台

超、凢脫俗 2022-01-27 17:43 569阅读 0赞

主机(服务端Server): `elaticsearch`, `kibana`, `fluentd aggregator` 安装在这台机器上

IP:`192.168.1.5`

节点机(客户端Client):

*  Node1: IP: `192.168.1.6` Nginx Server
 *  Nodel2: IP: `192.168.1.7` django Server

### 注意：1、2、3步服务端和客户端都要安装 ###

#### 1. 首先安装ruby ####

因为系统自带的ruby版本太低, 先删除掉, 然后安装依赖软件  
安装依赖软件  
`yum -y install zlib-devel libcurl-devel`  
接着安装yaml

tar xvf /soft/yaml-0.1.4.tar.gz   # 官网下载的
    cd yaml-0.1.4
    ./configure&&make&&make install

安装ruby

tar xvf /soft/ruby-1.9.3-p484.tar.gz
    cd ruby-1.9.3-p484
    ./configure --prefix=/usr/local/ruby --enable-shared --disable-install-doc --with-opt-dir=/usr/local/ && make &&make install

#### 2. 安装td-agent ####

下载地址:

`http://174.129.37.216/redhat/x86_64/td-agent-1.1.17-0.x86_64.rpm`

`http://174.129.37.216/redhat/x86_64/td-libyaml-0.1.4-1.x86_64.rpm`

软件包放在/soft下  
`yum -y install /soft/td-*`

#### 3. 安装td-agent插件 ####

删除自带的ruby源(指向国外的服务器)  
`/usr/lib64/fluent/ruby/bin/fluent-gem sources --remove http://rubygems.org/`

添加淘宝的源

`/usr/lib64/fluent/ruby/bin/fluent-gem sources --http-proxy http://172.17.17.132:235 -a http://ruby.taobao.org/`

安装secure-forward插件

`/usr/lib64/fluent/ruby/bin/fluent-gem install --http-proxy http://172.17.17.132:235 fluent-plugin-secure-forward`

#### 4. 安装elasticsearch(安装在服务端, 需要JDK1.7) ####

去官网下载RPM包

`rpm -ivh /soft/elasticsearch-1.3.2.noarch.rpm`

启动

`/etc/init.d/elasticsearch start`

#### 5. 安装kibana(安装在服务端) ####

去官网下载压缩包, 直接放在网站目录下(nginx和apache都可以), 里面都是静态页面

#### 6. 配置fluentd aggregator(安装在服务端) ####

默认打开的是24224端口, 请确保防火墙已经开启此端口  
安装fluent-plugin-elasticsearch插件

`/usr/lib64/fluent/ruby/bin/fluent-gem install --http-proxy http://172.17.17.132:235 fluent-plugin-elasticsearch`

开始编辑td-agent配置文件: /etc/td-agent/td-agent.conf

# Listen to incoming data over SSL 这是一种安装的连接, 必须密码才能访问服务传递数据, 不写默认是24224端口去接收数据
    <source>
      type secure_forward
      shared_key 123456 #密码设置，随便设，保持两边一致就行
      self_hostname localhost
      cert_auto_generate yes
    </source>
    
    # Store Data in Elasticsearch   将端口接收到的数据存入elasticsearch
    <match *.**>
      type copy
      <store>
        type elasticsearch
        host localhost  
        port 9200
        include_tag_key true
        tag_key log_name
        logstash_format true
        flush_interval 1s
      </store>
    </match>

#### 7. 抓取tornado日志 ####

配置客户端的td-agent

# Input
    <source>
      type forward
    </source>
    
    # Input
    <source>
      type tcp
      tag u.access.internal
      port 5170
      format /^(?<time>[^\"]*),[^ ]* - (?<status>[^ ]*) (?<method>[^ ]*) (?<request>[^ ]*) \((?<ip>[^ ]*)\) (?<u_time>[^m]*)ms (?<tour_id>[^ ]*) (?<xforwarded_for>[^ ]*)/
      types u_time:float,status:integer
      time_format %Y-%m-%d %H:%M:%S
    </source>
    
    # Log Forwarding
    <match *.**>
      type secure_forward
      shared_key 123456 #密码设置
      self_hostname localhost
      <server>
        host 192.168.1.5
      </server>
    
      # use file buffer to buffer events log on disks
      buffer_type file
      buffer_path /var/log/td-agent/*.buffer
    
      # use longer flush_interval to reduce CPU usage
      flush_interval 2s
    </match>