flume的source, channel, sink 列表

Love The Way You Lie 2022-08-22 00:28 153阅读 0赞

*  Flume Source

<table style="margin:1.5em 0px; padding:0px; border:thin solid rgb(134,134,134); outline:0px; font-size:15px; vertical-align:baseline; border-collapse:collapse; border-spacing:0px; width:990px; color:rgb(68,68,68); font-family:Verdana,Geneva,sans-serif; line-height:25.5px"> 
 <tbody style="margin:0px; padding:0px; border:0px; outline:0px; vertical-align:baseline"> 
 <tr style="margin:0px; padding:0px; border:0px; outline:0px; vertical-align:baseline"> 
 <td style="margin:0px; padding:5px; border:thin solid rgb(134,134,134); outline:0px; font-size:15px; vertical-align:top; width:187px; text-align:center"> Source类型</td> 
 <td style="margin:0px; padding:5px; border:thin solid rgb(134,134,134); outline:0px; font-size:15px; vertical-align:top; width:781px; text-align:center"> 说明</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; border:0px; outline:0px; vertical-align:baseline"> 
 <td style="margin:0px; padding:5px; border:thin solid rgb(134,134,134); outline:0px; font-size:15px; vertical-align:top; width:187px"> Avro Source</td> 
 <td style="margin:0px; padding:5px; border:thin solid rgb(134,134,134); outline:0px; font-size:15px; vertical-align:top; width:781px"> 支持Avro协议（实际上是Avro RPC），内置支持</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; border:0px; outline:0px; vertical-align:baseline"> 
 <td style="margin:0px; padding:5px; border:thin solid rgb(134,134,134); outline:0px; font-size:15px; vertical-align:top; width:187px"> Thrift Source</td> 
 <td style="margin:0px; padding:5px; border:thin solid rgb(134,134,134); outline:0px; font-size:15px; vertical-align:top; width:781px"> 支持Thrift协议，内置支持</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; border:0px; outline:0px; vertical-align:baseline"> 
 <td style="margin:0px; padding:5px; border:thin solid rgb(134,134,134); outline:0px; font-size:15px; vertical-align:top; width:187px"> Exec Source</td> 
 <td style="margin:0px; padding:5px; border:thin solid rgb(134,134,134); outline:0px; font-size:15px; vertical-align:top; width:781px"> 基于Unix的command在标准输出上生产数据</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; border:0px; outline:0px; vertical-align:baseline"> 
 <td style="margin:0px; padding:5px; border:thin solid rgb(134,134,134); outline:0px; font-size:15px; vertical-align:top; width:187px"> JMS Source</td> 
 <td style="margin:0px; padding:5px; border:thin solid rgb(134,134,134); outline:0px; font-size:15px; vertical-align:top; width:781px"> 从JMS系统（消息、主题）中读取数据，ActiveMQ已经测试过</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; border:0px; outline:0px; vertical-align:baseline"> 
 <td style="margin:0px; padding:5px; border:thin solid rgb(134,134,134); outline:0px; font-size:15px; vertical-align:top; width:187px"> Spooling Directory Source</td> 
 <td style="margin:0px; padding:5px; border:thin solid rgb(134,134,134); outline:0px; font-size:15px; vertical-align:top; width:781px"> 监控指定目录内数据变更</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; border:0px; outline:0px; vertical-align:baseline"> 
 <td style="margin:0px; padding:5px; border:thin solid rgb(134,134,134); outline:0px; font-size:15px; vertical-align:top; width:187px"> Twitter 1% firehose Source</td> 
 <td style="margin:0px; padding:5px; border:thin solid rgb(134,134,134); outline:0px; font-size:15px; vertical-align:top; width:781px"> 通过API持续下载Twitter数据，试验性质</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; border:0px; outline:0px; vertical-align:baseline"> 
 <td style="margin:0px; padding:5px; border:thin solid rgb(134,134,134); outline:0px; font-size:15px; vertical-align:top; width:187px"> Netcat Source</td> 
 <td style="margin:0px; padding:5px; border:thin solid rgb(134,134,134); outline:0px; font-size:15px; vertical-align:top; width:781px"> 监控某个端口，将流经端口的每一个文本行数据作为Event输入</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; border:0px; outline:0px; vertical-align:baseline"> 
 <td style="margin:0px; padding:5px; border:thin solid rgb(134,134,134); outline:0px; font-size:15px; vertical-align:top; width:187px"> Sequence Generator Source</td> 
 <td style="margin:0px; padding:5px; border:thin solid rgb(134,134,134); outline:0px; font-size:15px; vertical-align:top; width:781px"> 序列生成器数据源，生产序列数据</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; border:0px; outline:0px; vertical-align:baseline"> 
 <td style="margin:0px; padding:5px; border:thin solid rgb(134,134,134); outline:0px; font-size:15px; vertical-align:top; width:187px"> Syslog Sources</td> 
 <td style="margin:0px; padding:5px; border:thin solid rgb(134,134,134); outline:0px; font-size:15px; vertical-align:top; width:781px"> 读取syslog数据，产生Event，支持UDP和TCP两种协议</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; border:0px; outline:0px; vertical-align:baseline"> 
 <td style="margin:0px; padding:5px; border:thin solid rgb(134,134,134); outline:0px; font-size:15px; vertical-align:top; width:187px"> HTTP Source</td> 
 <td style="margin:0px; padding:5px; border:thin solid rgb(134,134,134); outline:0px; font-size:15px; vertical-align:top; width:781px"> 基于HTTP POST或GET方式的数据源，支持JSON、BLOB表示形式</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; border:0px; outline:0px; vertical-align:baseline"> 
 <td style="margin:0px; padding:5px; border:thin solid rgb(134,134,134); outline:0px; font-size:15px; vertical-align:top; width:187px"> Legacy Sources</td> 
 <td style="margin:0px; padding:5px; border:thin solid rgb(134,134,134); outline:0px; font-size:15px; vertical-align:top; width:781px"> 兼容老的Flume OG中Source（0.9.x版本）</td> 
 </tr> 
 </tbody> 
</table>

*  Flume Channel

*  Flume Sink

<table style="margin:1.5em 0px; padding:0px; border:thin solid rgb(134,134,134); outline:0px; font-size:15px; vertical-align:baseline; border-collapse:collapse; border-spacing:0px; width:990px; color:rgb(68,68,68); font-family:Verdana,Geneva,sans-serif; line-height:25.5px"> 
 <tbody style="margin:0px; padding:0px; border:0px; outline:0px; vertical-align:baseline; background:transparent"> 
 <tr style="margin:0px; padding:0px; border:0px; outline:0px; vertical-align:baseline; background:transparent"> 
 <td style="margin:0px; padding:5px; border:thin solid rgb(134,134,134); outline:0px; font-size:15px; vertical-align:top; width:187px; text-align:center; background:transparent"> Sink类型</td> 
 <td style="margin:0px; padding:5px; border:thin solid rgb(134,134,134); outline:0px; font-size:15px; vertical-align:top; width:781px; text-align:center; background:transparent"> 说明</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; border:0px; outline:0px; vertical-align:baseline; background:transparent"> 
 <td style="margin:0px; padding:5px; border:thin solid rgb(134,134,134); outline:0px; font-size:15px; vertical-align:top; width:187px; background:transparent"> HDFS Sink</td> 
 <td style="margin:0px; padding:5px; border:thin solid rgb(134,134,134); outline:0px; font-size:15px; vertical-align:top; width:781px; background:transparent"> 数据写入HDFS</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; border:0px; outline:0px; vertical-align:baseline; background:transparent"> 
 <td style="margin:0px; padding:5px; border:thin solid rgb(134,134,134); outline:0px; font-size:15px; vertical-align:top; width:187px; background:transparent"> Logger Sink</td> 
 <td style="margin:0px; padding:5px; border:thin solid rgb(134,134,134); outline:0px; font-size:15px; vertical-align:top; width:781px; background:transparent"> 数据写入日志文件</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; border:0px; outline:0px; vertical-align:baseline; background:transparent"> 
 <td style="margin:0px; padding:5px; border:thin solid rgb(134,134,134); outline:0px; font-size:15px; vertical-align:top; width:187px; background:transparent"> Avro Sink</td> 
 <td style="margin:0px; padding:5px; border:thin solid rgb(134,134,134); outline:0px; font-size:15px; vertical-align:top; width:781px; background:transparent"> 数据被转换成Avro Event，然后发送到配置的RPC端口上</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; border:0px; outline:0px; vertical-align:baseline; background:transparent"> 
 <td style="margin:0px; padding:5px; border:thin solid rgb(134,134,134); outline:0px; font-size:15px; vertical-align:top; width:187px; background:transparent"> Thrift Sink</td> 
 <td style="margin:0px; padding:5px; border:thin solid rgb(134,134,134); outline:0px; font-size:15px; vertical-align:top; width:781px; background:transparent"> 数据被转换成Thrift Event，然后发送到配置的RPC端口上</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; border:0px; outline:0px; vertical-align:baseline; background:transparent"> 
 <td style="margin:0px; padding:5px; border:thin solid rgb(134,134,134); outline:0px; font-size:15px; vertical-align:top; width:187px; background:transparent"> IRC Sink</td> 
 <td style="margin:0px; padding:5px; border:thin solid rgb(134,134,134); outline:0px; font-size:15px; vertical-align:top; width:781px; background:transparent"> 数据在IRC上进行回放</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; border:0px; outline:0px; vertical-align:baseline; background:transparent"> 
 <td style="margin:0px; padding:5px; border:thin solid rgb(134,134,134); outline:0px; font-size:15px; vertical-align:top; width:187px; background:transparent"> File Roll Sink</td> 
 <td style="margin:0px; padding:5px; border:thin solid rgb(134,134,134); outline:0px; font-size:15px; vertical-align:top; width:781px; background:transparent"> 存储数据到本地文件系统</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; border:0px; outline:0px; vertical-align:baseline; background:transparent"> 
 <td style="margin:0px; padding:5px; border:thin solid rgb(134,134,134); outline:0px; font-size:15px; vertical-align:top; width:187px; background:transparent"> Null Sink</td> 
 <td style="margin:0px; padding:5px; border:thin solid rgb(134,134,134); outline:0px; font-size:15px; vertical-align:top; width:781px; background:transparent"> 丢弃到所有数据</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; border:0px; outline:0px; vertical-align:baseline; background:transparent"> 
 <td style="margin:0px; padding:5px; border:thin solid rgb(134,134,134); outline:0px; font-size:15px; vertical-align:top; width:187px; background:transparent"> HBase Sink</td> 
 <td style="margin:0px; padding:5px; border:thin solid rgb(134,134,134); outline:0px; font-size:15px; vertical-align:top; width:781px; background:transparent"> 数据写入HBase数据库</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; border:0px; outline:0px; vertical-align:baseline; background:transparent"> 
 <td style="margin:0px; padding:5px; border:thin solid rgb(134,134,134); outline:0px; font-size:15px; vertical-align:top; width:187px; background:transparent"> Morphline Solr Sink</td> 
 <td style="margin:0px; padding:5px; border:thin solid rgb(134,134,134); outline:0px; font-size:15px; vertical-align:top; width:781px; background:transparent"> 数据发送到Solr搜索服务器（集群）</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; border:0px; outline:0px; vertical-align:baseline; background:transparent"> 
 <td style="margin:0px; padding:5px; border:thin solid rgb(134,134,134); outline:0px; font-size:15px; vertical-align:top; width:187px; background:transparent"> ElasticSearch Sink</td> 
 <td style="margin:0px; padding:5px; border:thin solid rgb(134,134,134); outline:0px; font-size:15px; vertical-align:top; width:781px; background:transparent"> 数据发送到Elastic Search搜索服务器（集群）</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; border:0px; outline:0px; vertical-align:baseline; background:transparent"> 
 <td style="margin:0px; padding:5px; border:thin solid rgb(134,134,134); outline:0px; font-size:15px; vertical-align:top; width:187px; background:transparent"> Kite Dataset Sink</td> 
 <td style="margin:0px; padding:5px; border:thin solid rgb(134,134,134); outline:0px; font-size:15px; vertical-align:top; width:781px; background:transparent"> 写数据到Kite Dataset，试验性质的</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; border:0px; outline:0px; vertical-align:baseline; background:transparent"> 
 <td style="margin:0px; padding:5px; border:thin solid rgb(134,134,134); outline:0px; font-size:15px; vertical-align:top; width:187px; background:transparent"> Custom Sink</td> 
 <td style="margin:0px; padding:5px; border:thin solid rgb(134,134,134); outline:0px; font-size:15px; vertical-align:top; width:781px; background:transparent"> 自定义Sink实现 </td> 
 </tr> 
 </tbody> 
</table>