flume报错：Sinks are likely not keeping up with sources, or the buffer size is too tight

￡神魔★判官ぃ 2021-06-24 14:37 303阅读 0赞

使用flume收集日志，在agent的日志中发现如下错误信息：

Caused by: org.apache.flume.ChannelException: Space for commit to queue couldn't be acquired Sinks are likely not keeping up with sources, or the buffer size is too tight
            at org.apache.flume.channel.MemoryChannel$MemoryTransaction.doCommit(MemoryChannel.java:126)
            at org.apache.flume.channel.BasicTransactionSemantics.commit(BasicTransactionSemantics.java:151)
            at org.apache.flume.channel.ChannelProcessor.processEventBatch(ChannelProcessor.java:192)
            ... 28 more

解决方法：

1）修改flume-ng agen的堆内存：

vim flume-env.sh
    export JAVA_OPTS="-Xms1024m -Xmx1024m"

默认是20m

2）修改channel的配置信息：

agent1.channels.mc1.capacity = 1000
    agent1.channels.mc1.keep-alive = 60

解释：

<table style="border-spacing:0px; border-collapse:collapse; width:815px; max-width:100%; margin-bottom:20px; border:1px solid rgb(221,221,221); color:rgb(51,51,51); font-family:&quot;Helvetica Neue&quot;,Helvetica,Arial,sans-serif; font-size:14px"> 
 <tbody style=""> 
  <tr style=""> 
   <td style="padding:8px; line-height:1.42857; vertical-align:top; border:1px solid rgb(221,221,221)"> channels</td> 
   <td style="padding:8px; line-height:1.42857; vertical-align:top; border:1px solid rgb(221,221,221)"> -</td> 
   <td style="padding:8px; line-height:1.42857; vertical-align:top; border:1px solid rgb(221,221,221)"> &nbsp;</td> 
  </tr> 
  <tr style=""> 
   <td style="padding:8px; line-height:1.42857; vertical-align:top; border:1px solid rgb(221,221,221); background-color:rgb(249,249,249)"> type</td> 
   <td style="padding:8px; line-height:1.42857; vertical-align:top; border:1px solid rgb(221,221,221); background-color:rgb(249,249,249)"> -</td> 
   <td style="padding:8px; line-height:1.42857; vertical-align:top; border:1px solid rgb(221,221,221); background-color:rgb(249,249,249)"> 组件类型名称必须是<code style="font-family:Menlo,Monaco,Consolas,&quot;Courier New&quot;,monospace; font-size:12.6px; padding:2px 4px; color:rgb(199,37,78); background-color:rgb(249,242,244)">memory</code></td> 
  </tr> 
  <tr style=""> 
   <td style="padding:8px; line-height:1.42857; vertical-align:top; border:1px solid rgb(221,221,221)"> capacity</td> 
   <td style="padding:8px; line-height:1.42857; vertical-align:top; border:1px solid rgb(221,221,221)"> 100</td> 
   <td style="padding:8px; line-height:1.42857; vertical-align:top; border:1px solid rgb(221,221,221)"> 存储在 Channel 当中的最大 events 数</td> 
  </tr> 
  <tr style=""> 
   <td style="padding:8px; line-height:1.42857; vertical-align:top; border:1px solid rgb(221,221,221); background-color:rgb(249,249,249)"> transactionCapacity</td> 
   <td style="padding:8px; line-height:1.42857; vertical-align:top; border:1px solid rgb(221,221,221); background-color:rgb(249,249,249)"> 100</td> 
   <td style="padding:8px; line-height:1.42857; vertical-align:top; border:1px solid rgb(221,221,221); background-color:rgb(249,249,249)"> 同时刻从Source 获取，或发送到 Sink 的最大 events 数</td> 
  </tr> 
  <tr style=""> 
   <td style="padding:8px; line-height:1.42857; vertical-align:top; border:1px solid rgb(221,221,221)"> keep-alive</td> 
   <td style="padding:8px; line-height:1.42857; vertical-align:top; border:1px solid rgb(221,221,221)"> 3</td> 
   <td style="padding:8px; line-height:1.42857; vertical-align:top; border:1px solid rgb(221,221,221)"> 添加或删除一个 event 超时的秒数</td> 
  </tr> 
 </tbody> 
</table>

keep-alive"这个参数用来控制channel满时影响source的发送，channel空时影响sink的消费，就是等待时间，默认是3s，超过这个时间就甩异常，一般不需配置，但是有些情况很有用，比如你得场景是每分钟开头集中发一次数据，这时每分钟的开头量可能比较大，后面会越来越小，这时你可以调大这个参数，不至于出现channel满了得情况；