Flume-负载均衡和故障转移案例

r囧r小猫 2022-10-04 13:57 146阅读 0赞

### 负载均衡和故障转移 ###

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2NsZWFybHhq_size_16_color_FFFFFF_t_70]

使用Flume监控一个端口，其sink组中的sink分别对接Flume2和Flume3，采用FailoverSinkProcessor，实现故障转移的功能。其中Flume2和Flume3的Sink直接采用Logger方便查看和测试。

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2NsZWFybHhq_size_16_color_FFFFFF_t_70 1]

FailoverSinkProcessor：故障转移处理器维护了一个发送Event失败的sink的列表，保证有一个sink是可用的来发送Event。

配置及说明如下：

故障转移机制的工作原理是将故障sink降级到一个池中，在池中为它们分配冷却期（超时时间），默认30s。 在到达冷却时间之后，它将恢复到实时池。在冷却期中的sink不被考虑使用，只有恢复到实时池中的才能通过优先级被考虑使用。

sink具有与之相关的优先级，数值越大，优先级越高。 如果在发送Event时Sink发生故障，会继续尝试下一个具有最高优先级的sink。 例如，有100、80、50三个优先级的sink，优先级100的sink发生了故障则使用优先级有80的sink。如果未指定优先级，则根据配置中的顺序来选取。

要使用故障转移选择器，不仅要设置sink组的选择器为failover，还有为每一个sink设置一个唯一的优先级数值。 可以使用 *maxpenalty* 属性设置故障转移时间的上限（毫秒）。

![在这里插入图片描述][2021071516112221.png]

Flume1配置文件如下：

# Name the components on this agent
    a1.sources = r1
    a1.sinks = k1 k2
    a1.channels = c1
    
    # Describe/configure the source
    a1.sources.r1.type = netcat
    a1.sources.r1.bind = localhost
    a1.sources.r1.port = 44444
    
    # Describe the sink
    a1.sinks.k1.type = avro
    a1.sinks.k1.hostname = hadoop113
    a1.sinks.k1.port = 4441
    a1.sinks.k2.type = avro
    a1.sinks.k2.hostname = hadoop113
    a1.sinks.k2.port = 4442
    
    # Sink Group
    # channel对多个sink，默认是负载均衡，且是轮询的
    a1.sinkgroups = g1
    a1.sinkgroups.g1.sinks = k1 k2
    a1.sinkgroups.g1.processor.type = failover
    a1.sinkgroups.g1.processor.priority.k1 = 5
    a1.sinkgroups.g1.processor.priority.k2 = 10
    a1.sinkgroups.g1.processor.maxpenalty = 10000
    
    # Use a channel which buffers events in memory
    a1.channels.c1.type = memory
    a1.channels.c1.capacity = 1000
    a1.channels.c1.transactionCapacity = 100
    
    # Bind the source and sink to the channel
    a1.sources.r1.channels = c1
    a1.sinks.k1.channel = c1
    a1.sinks.k2.channel = c1

Flume2配置文件如下：

# Name the components on this agent
    a2.sources = r2
    a2.sinks = k2
    a2.channels = c2
    
    # Describe/configure the source
    a2.sources.r2.type = avro
    a2.sources.r2.bind = hadoop113
    a2.sources.r2.port = 4441
    
    # Describe the sink
    a2.sinks.k2.type = logger
    
    # Use a channel which buffers events in memory
    a2.channels.c2.type = memory
    a2.channels.c2.capacity = 1000
    a2.channels.c2.transactionCapacity = 100
    
    # Bind the source and sink to the channel
    a2.sources.r2.channels = c2
    a2.sinks.k2.channel = c2

Flume3配置文件如下：

# Name the components on this agent
    a3.sources = r2
    a3.sinks = k2
    a3.channels = c2
    
    # Describe/configure the source
    a3.sources.r2.type = avro
    a3.sources.r2.bind = hadoop113
    a3.sources.r2.port = 4442
    
    # Sink
    a3.sinks.k2.type = logger
    
    
    # Use a channel which buffers events in memory
    a3.channels.c2.type = memory
    a3.channels.c2.capacity = 1000
    a3.channels.c2.transactionCapacity = 100
    
    # Bind the source and sink to the channel
    a3.sources.r2.channels = c2
    a3.sinks.k2.channel = c2

通过`nc localhost 44444`连接到44444端口，依次输入hello、word、hahahah，结果全部在Flume3中输出了。  
![在这里插入图片描述][20210715161107217.png]

此时我们关闭Flume3，依次输入test、failover，结果全在Flume中输出了。

![在这里插入图片描述][20210715161100178.png]

说明故障转移成功了。

负载均衡的话只需要将Sink组的SinkProcessor设置成load\_balance即可，其他的和故障转移一模一样

# Sink Group
    # channel对多个sink，不设置的话默认是负载均衡，且是轮询的
    a1.sinkgroups = g1
    a1.sinkgroups.g1.sinks = k1 k2
    a1.sinkgroups.g1.processor.type = load_balance
    a1.sinkgroups.g1.processor.backoff = true
    ## 这里设置成random
    a1.sinkgroups.g1.processor.selector = random

运行结果如下：

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2NsZWFybHhq_size_16_color_FFFFFF_t_70 2]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2NsZWFybHhq_size_16_color_FFFFFF_t_70]: /images/20221004/b98b576d93f240fd8222092fd3213083.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2NsZWFybHhq_size_16_color_FFFFFF_t_70 1]: /images/20221004/0058876d4cce48448c42c46be9caaf50.png
[2021071516112221.png]: /images/20221004/650d582f1f30459c8001b0bf830087e9.png
[20210715161107217.png]: /images/20221004/06d530bac24e4bc987bee40255bd3d7e.png
[20210715161100178.png]: /images/20221004/0b238063b38f4feaa44e593dd2f054c9.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2NsZWFybHhq_size_16_color_FFFFFF_t_70 2]: /images/20221004/70d350180f8149a88f4e92f3edab9dd0.png