apache flume agent安装

﹏ヽ暗。殇╰゛Y 2022-08-22 03:49 156阅读 0赞

##  ##

# 1、Flume Agent安装（采用spoolDir模式以便获取系统、应用等日志信息） #

注：使用jyapp用户安装

一台虚拟机部署了多个JAVA应用同时需要部署多个flume-agent进行监控时，

需要调整如下配置文件：

a    flume-agent/conf/app.conf中的spool\_Dir参数

b    flume-agent/conf/flume-env.sh中的jmxremote.port参数

c    logback.xml中fileNamePattern参数

d    flume\_spooldir.sh中的dir参数

## 1.1 安装环境 ##

系统:

CentOS release 6.6

软件:

flume-agent.tar.gz

flume-agent.tar.gz和flume\_spooldir.sh文件下载路径：ftp://172.21.100.61/pro/flume/flume-agent

## 1.2 安装步骤 ##

### 1.2.1 部署Flume Agent ###

具体脚本（jyapp用户）：

<table style="border-collapse:collapse; border-spacing:0px; width:1843px; max-width:100%; margin-bottom:20px; color:rgb(51,51,51); font-family:'Helvetica Neue',Helvetica,Arial,sans-serif; font-size:14px; line-height:20px"> 
 <tbody style=""> 
 <tr style=""> 
 <td style="padding:8px; line-height:1.42857143; vertical-align:top; border-top-width:1px; border-top-style:solid; border-top-color:rgb(221,221,221)"> 
 <div style=""> 
 cd &nbsp;/home/jyapp 
 mkdir &nbsp;flume_logs 
 tar &nbsp;-zxvf &nbsp;flume-agent.tar.gz 
 <div style="height:1em"></div> 
 cd &nbsp;flume-agent 
 <div style="height:1em"></div> 
 #修改flume环境变量 
 vi &nbsp;conf/flume-env.sh&nbsp;&nbsp;&nbsp; 
 具体配置解析：（注意：红色需修改） 
 ------------------------------------------------------ 
 export JAVA_OPTS="-Xms100m -Xmx100m &nbsp;-Dflume.root.logger=INFO,LOGFILE &nbsp;-Dflume.monitoring.type=http &nbsp;-Dflume.monitoring.port=34545 &nbsp;-Dcom.sun.management.jmxremote -Djava.rmi.server.hostname=本机IP&nbsp;&nbsp;-Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.authenticate=false &nbsp;-Dcom.sun.management.jmxremote.ssl=false" 
 ------------------------------------------------------------------------------------- 
 &nbsp; 
 #修改agent脚本 
 vi &nbsp;conf/app.conf&nbsp;&nbsp; 
 具体配置解析：（注意：红色需修改） 
 ------------------------------------------------------ 
 app.sources = r1 
 app.sinks = k1 k2 
 app.channels = c1 
 &nbsp; 
 app.sources.r1.type = spooldir 
 app.sources.r1.spoolDir = /home/jyapp/flume_logs &nbsp;&nbsp;&nbsp;#定义监控的日志目录 
 app.sources.r1.deletePolicy = immediate 
 app.sources.r1.ignorePattern = ^(.)*\\.tmp$ &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;#忽略.tmp文件 
 app.sources.r1.deserializer = MULTILINE &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;#使用多行反序列化器 
 app.sources.r1.deserializer.maxLineLength = 60000 
 app.sources.r1.interceptors = i1 i2 i3 
 app.sources.r1.interceptors.i1.type = timestamp &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;#设置时间戳 
 app.sources.r1.interceptors.i2.type = static 
 app.sources.r1.interceptors.i2.key = flume.client.ip 
 app.sources.r1.interceptors.i2.value =&nbsp;客户机IP地址 &nbsp;&nbsp;#设置IP地址 
 app.sources.r1.interceptors.i3.type = static 
 app.sources.r1.interceptors.i3.key = flume.client.system.flag&nbsp;&nbsp; 
 app.sources.r1.interceptors.i3.value =&nbsp;客户机系统标示 &nbsp;&nbsp;#设置系统标示，如S001 
 &nbsp; 
 app.channels.c1.type = memory 
 app.channels.c1.capacity = 1000 
 app.channels.c1.transactionCapacity = 1000 
 app.channels.c1.keep-alive = 3 
 app.channels.c1.byteCapacityBufferPercentage = 30 
 app.channels.c1.byteCapacity = 104857600 
 &nbsp; 
 app.sinks.k1.type = avro app.sinks.k1.hostname = 172.19.100.92 app.sinks.k1.port = 44466 app.sinks.k2.type = avro app.sinks.k2.hostname = 172.19.100.102 app.sinks.k2.port = 44466&nbsp; 
 app.sources.r1.channels = c1 
 app.sinks.k1.channel = c1 
 app.sinks.k2.channel = c1 
 <div style="height:1em"></div> 
 app.sinkgroups = g1 app.sinkgroups.g1.sinks = k1 k2 app.sinkgroups.g1.processor.type = load_balance app.sinkgroups.g1.processor.backoff = true app.sinkgroups.g1.processor.selector = round_robin 
 ------------------------------------------------------------------------------------- 
 批量替换命令参考： 
 #修改flume-env.sh 
 sed -i 's/172.18.100.159/172.19.100.73/g;s/S000/S001/g' &nbsp;/home/jyapp/flume-agent/conf/flume-env.sh 
 #修改app.conf 
 sed -i 's/172.18.100.159/172.19.100.73/g;s/S000/S001/g' &nbsp;/home/jyapp/flume-agent/conf/app.conf 
 <div style="height:1em"></div> 
 后台启动agent 
 cd &nbsp;/home/jyapp/flume-agent 
 bin/flume-ng &nbsp;agent &nbsp;--conf conf &nbsp;--conf-file &nbsp;conf/app.conf &nbsp;--name app &amp; 
 &nbsp; 
 停止agent，使用kill命令 
 </div> </td> 
 </tr> 
 </tbody> 
</table>