Flume - 故障转移、负载均衡

快来打我* 2023-06-09 10:27 74阅读 0赞
  • 故障转移

概述：

故障转移机制的工作方式是将失败的sink放到一个池中，并在池中为它们分配一段冷冻期，在重试之前随着连续的失败而增加。一个sink成功发送event后，将其恢复到活动池。sink有一个与它们相关联的优先级，数字越大表示优先级越高。如果一个sink在发送event时失败，则下一个具有最高优先级的sink将被尝试用于发送事件。

watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0JsZXNzaW5nWFJZ_size_16_color_FFFFFF_t_70

实例：

  1. # hadoop105
  2. # Name the components on this agent
  3. a1.sources = r1
  4. a1.sinks = k1 k2
  5. a1.channels = c1
  6. a1.sinkgroups = g1
  8. # Describe/configure the source
  9. a1.sources.r1.type = netcat
  10. a1.sources.r1.bind = localhost
  11. a1.sources.r1.port = 44444
  13. # Channel
  14. a1.channels.c1.type = memory
  15. a1.channels.c1.capacity = 1000
  16. a1.channels.c1.transactionCapacity = 100
  18. # Sink
  19. a1.sinks.k1.type = avro
  20. a1.sinks.k1.hostname = hadoop106
  21. a1.sinks.k1.port = 4141
  23. a1.sinks.k2.type = avro
  24. a1.sinks.k2.hostname = hadoop107
  25. a1.sinks.k2.port = 4142
  27. #Sink Group
  28. a1.sinkgroups.g1.sinks = k1 k2
  29. a1.sinkgroups.g1.processor.type = failover
  30. a1.sinkgroups.g1.processor.priority.k1 = 5
  31. a1.sinkgroups.g1.processor.priority.k2 = 10
  32. a1.sinkgroups.g1.processor.maxpenalty = 10000
  34. #Bind
  35. a1.sources.r1.channels = c1
  36. a1.sinks.k1.channel = c1
  37. a1.sinks.k2.channel = c1
  38. ----------------------------------------------------------------------
  39. # hadoop106
  40. #Name
  41. a2.sources = r1
  42. a2.channels = c1
  43. a2.sinks = k1
  45. #Source
  46. a2.sources.r1.type = avro
  47. a2.sources.r1.bind = hadoop106
  48. a2.sources.r1.port = 4141
  50. #Channel
  51. a2.channels.c1.type = memory
  52. a2.channels.c1.capacity = 1000
  53. a2.channels.c1.transactionCapacity = 100
  55. #Sink
  56. a2.sinks.k1.type = logger
  58. #Bind
  59. a2.sources.r1.channels = c1
  60. a2.sinks.k1.channel = c1
  61. ----------------------------------------------------------------------
  62. # hadoop107
  63. #Name
  64. a3.sources = r1
  65. a3.channels = c1
  66. a3.sinks = k1
  68. #Source
  69. a3.sources.r1.type = avro
  70. a3.sources.r1.bind = hadoop107
  71. a3.sources.r1.port = 4142
  73. #Channel
  74. a3.channels.c1.type = memory
  75. a3.channels.c1.capacity = 1000
  76. a3.channels.c1.transactionCapacity = 100
  78. #Sink
  79. a3.sinks.k1.type = logger
  81. #Bind
  82. a3.sources.r1.channels = c1
  83. a3.sinks.k1.channel = c1

参考文章链接：
文章1：http://flume.apache.org/releases/content/1.9.0/FlumeUserGuide.html
文章2：https://blog.51cto.com/jackwxh/1906893

  • 负载均衡

概述：

source中的event流经channel，进入sink group，在sink group中根据负载算法（round_robin、random）选择sink，然后选择不同机器上的agent实现负载均衡。

watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0JsZXNzaW5nWFJZ_size_16_color_FFFFFF_t_70 1

  1. # hadoop105
  2. # Name the components on this agent
  3. a1.sources = r1
  4. a1.sinks = k1 k2
  5. a1.channels = c1
  6. a1.sinkgroups = g1
  8. # Describe/configure the source
  9. a1.sources.r1.type = netcat
  10. a1.sources.r1.bind = localhost
  11. a1.sources.r1.port = 44444
  13. # Channel
  14. a1.channels.c1.type = memory
  15. a1.channels.c1.capacity = 1000
  16. a1.channels.c1.transactionCapacity = 100
  18. # Sink
  19. a1.sinks.k1.type = avro
  20. a1.sinks.k1.hostname = hadoop106
  21. a1.sinks.k1.port = 4141
  23. a1.sinks.k2.type = avro
  24. a1.sinks.k2.hostname = hadoop107
  25. a1.sinks.k2.port = 4142
  27. #Sink Group
  28. a1.sinkgroups.g1.sinks = k1 k2
  29. a1.sinkgroups.g1.processor.type = load_balance
  30. a1.sinkgroups.g1.processor.backoff = true
  31. a1.sinkgroups.g1.processor.selector = random
  33. #Bind
  34. a1.sources.r1.channels = c1
  35. a1.sinks.k1.channel = c1
  36. a1.sinks.k2.channel = c1
  37. ----------------------------------------------------------------------
  38. # hadoop106
  39. #Name
  40. a2.sources = r1
  41. a2.channels = c1
  42. a2.sinks = k1
  44. #Source
  45. a2.sources.r1.type = avro
  46. a2.sources.r1.bind = hadoop106
  47. a2.sources.r1.port = 4141
  49. #Channel
  50. a2.channels.c1.type = memory
  51. a2.channels.c1.capacity = 1000
  52. a2.channels.c1.transactionCapacity = 100
  54. #Sink
  55. a2.sinks.k1.type = logger
  57. #Bind
  58. a2.sources.r1.channels = c1
  59. a2.sinks.k1.channel = c1
  60. ----------------------------------------------------------------------
  61. # hadoop107
  62. #Name
  63. a3.sources = r1
  64. a3.channels = c1
  65. a3.sinks = k1
  67. #Source
  68. a3.sources.r1.type = avro
  69. a3.sources.r1.bind = hadoop107
  70. a3.sources.r1.port = 4142
  72. #Channel
  73. a3.channels.c1.type = memory
  74. a3.channels.c1.capacity = 1000
  75. a3.channels.c1.transactionCapacity = 100
  77. #Sink
  78. a3.sinks.k1.type = logger
  80. #Bind
  81. a3.sources.r1.channels = c1
  82. a3.sinks.k1.channel = c1

参考文章地址：
文章1：http://flume.apache.org/releases/content/1.9.0/FlumeUserGuide.html
文章2：https://blog.csdn.net/silentwolfyh/article/details/51165804

发表评论

表情：
评论列表 （有 0 条评论，74人围观）

还没有评论，来说两句吧...

相关阅读

    相关 Flume ——故障转移

    Sink groups允许组织多个sink到一个实体上。 Sink processors能够提供在组内所有Sink之间实现负载均衡的能力，而且在失败的情况下能够进行故障转移从一

    傷城~傷城~/ 0 赞/ 353 阅读