大数据流处理平台的技术选型参考

àì夳堔傛蜴生んèń 2022-05-29 02:00 233阅读 0赞

##                              大数据流处理平台的技术选型参考 ##

选择太多，是一件好事情，不过也容易乱花渐欲迷人眼。倘若每个平台（技术）都去动手操练一下，似乎又太耗时间。通过阅读一些文档，可以帮我们快速做一次筛选。在将选择范围进一步缩小后，接下来就可以结合自己的应用场景去深入Spike，做深度的甄别，这是我做技术选型的一个方法。

技术没有最好，只有最适用。在做技术选型时，需要选择适合需求、适合项目类型、适合团队的技术。这是实用主义的判断，而非理想主义的追捧。若是在实用的技术选型中，再能点燃一些些技术上的情怀，那就perfect了！

###     ###

### 属性矩阵(Attributes Matrix) ###

我在《Apache下流处理项目巡览》（https://iamzhangyi.github.io/Apache-Streaming-Projects/）一文中翻译了Janakiram的这篇文章，介绍了Apache基金会下最主流的流处理项目。巧的是，我在InfoQ上又发现了Ian Hellstrom的文章（http://www.infoq.com/cn/articles/overview-of-apache-streaming-technology），他用一张图给出了非常棒的总结。

为了更好地阅读，我将这张图的内容转成如下的矩阵表。由于Ian的文章是2016年撰写的，我对其内容做了适度更新。

表一：流平台的质量属性

<table style="margin:0px 0px 10px; padding:0px; border-collapse:collapse; width:461px; max-width:100%; color:rgb(62,62,62); font-family:&quot;Hiragino Sans GB&quot;,&quot;Microsoft YaHei&quot;,Arial,sans-serif; font-size:16px; line-height:25.6px; widows:1; word-wrap:break-word!important"> 
 <thead style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important"> 
 <tr style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important"> 
 <th style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border-width:2px 1px 1px; border-style:solid; border-color:rgb(187,187,187) rgb(221,221,221) rgb(221,221,221); max-width:100%; background:rgb(247,247,247)"> 流平台</th> 
 <th style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border-width:2px 1px 1px; border-style:solid; border-color:rgb(187,187,187) rgb(221,221,221) rgb(221,221,221); max-width:100%; background:rgb(247,247,247)"> 当前版本</th> 
 <th style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border-width:2px 1px 1px; border-style:solid; border-color:rgb(187,187,187) rgb(221,221,221) rgb(221,221,221); max-width:100%; background:rgb(247,247,247)"> 主要推动者</th> 
 <th style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border-width:2px 1px 1px; border-style:solid; border-color:rgb(187,187,187) rgb(221,221,221) rgb(221,221,221); max-width:100%; background:rgb(247,247,247)"> Event Size</th> 
 <th style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border-width:2px 1px 1px; border-style:solid; border-color:rgb(187,187,187) rgb(221,221,221) rgb(221,221,221); max-width:100%; background:rgb(247,247,247)"> 消息传递保证</th> 
 <th style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border-width:2px 1px 1px; border-style:solid; border-color:rgb(187,187,187) rgb(221,221,221) rgb(221,221,221); max-width:100%; background:rgb(247,247,247)"> 状态管理</th> 
 </tr> 
 </thead> 
 <tbody style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important"> 
 <tr style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important"> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Flume</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> 1.8.0</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Apple, Cloudera</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> single</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> at least once</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> 事务型更新</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important"> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> NiFi</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> 1.5.0</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Hortonworks</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> single</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> at least once</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> 本地及分布式快照</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important"> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Gearpump</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> 0.8.4</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> single</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Intel, Lightbend</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> exactly once，若不需要支持容错，则为at least once</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> checkpoints</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important"> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Apex</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Apex Core 3.6.0, Apex Malhar 3.8.0</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Data Torrent</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> single</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> exactly once</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> checkpoints</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important"> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Kafka Streams</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> 1.0</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Confluent</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> single</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> at least once</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> 本地及分布式快照</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important"> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Spark Streaming</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> 2.2.1</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> AMPLab, Databricks</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> micro-batch</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> exactly once，若不需要支持容错，则为at least once</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> checkpoints</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important"> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Storm</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> 1.1.1</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Backtype, Twitter</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> single</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> at least once</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> record acknowledgements</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important"> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Samza</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> 0.14.0</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Linkedin</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> single</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> at least once</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> 本地快照，分布式快照则支持容错</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important"> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Flink</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> 1.4.0</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> dataArtisans</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> single</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> exactly once</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> 分布式快照</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important"> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Ignite Streaming</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> 2.3.0</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> GridGain</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> single</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> at least once</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> checkpoints</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important"> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Beam</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> 2.2.0</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Google</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> single</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> exactly once</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> 事务型更新</td> 
 </tr> 
 </tbody> 
</table>

表一（续）：流平台的质量属性

<table style="margin:0px 0px 10px; padding:0px; border-collapse:collapse; width:461px; max-width:100%; color:rgb(62,62,62); font-family:&quot;Hiragino Sans GB&quot;,&quot;Microsoft YaHei&quot;,Arial,sans-serif; font-size:16px; line-height:25.6px; widows:1; word-wrap:break-word!important"> 
 <thead style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important"> 
 <tr style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important"> 
 <th style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border-width:2px 1px 1px; border-style:solid; border-color:rgb(187,187,187) rgb(221,221,221) rgb(221,221,221); max-width:100%; background:rgb(247,247,247)"> 流平台</th> 
 <th style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border-width:2px 1px 1px; border-style:solid; border-color:rgb(187,187,187) rgb(221,221,221) rgb(221,221,221); max-width:100%; background:rgb(247,247,247)"> 容错</th> 
 <th style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border-width:2px 1px 1px; border-style:solid; border-color:rgb(187,187,187) rgb(221,221,221) rgb(221,221,221); max-width:100%; background:rgb(247,247,247)"> 处理顺序</th> 
 <th style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border-width:2px 1px 1px; border-style:solid; border-color:rgb(187,187,187) rgb(221,221,221) rgb(221,221,221); max-width:100%; background:rgb(247,247,247)"> 事件的优先级</th> 
 <th style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border-width:2px 1px 1px; border-style:solid; border-color:rgb(187,187,187) rgb(221,221,221) rgb(221,221,221); max-width:100%; background:rgb(247,247,247)"> Windowing</th> 
 <th style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border-width:2px 1px 1px; border-style:solid; border-color:rgb(187,187,187) rgb(221,221,221) rgb(221,221,221); max-width:100%; background:rgb(247,247,247)"> Back-pressure(背压）</th> 
 </tr> 
 </thead> 
 <tbody style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important"> 
 <tr style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important"> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Flume</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> yes(只针对file channel)</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> no</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> no</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> no</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> no</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important"> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> NiFi</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> yes</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> no</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> yes</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> no</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> yes</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important"> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Gearpump</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> yes</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> yes</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> programmable</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> time-based</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> yes</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important"> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Apex</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> yes</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> no</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> programmable</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> time-based</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> yes</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important"> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Kafka Streams</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> yes</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> yes</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> programmable</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> time-based</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> N/A</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important"> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Spark Streaming</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> yes</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> no</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> programmable</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> time-based</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> yes</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important"> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Storm</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> yes</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> yes</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> programmable</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> time-based, count-based</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> yes</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important"> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Samza</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> yes</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> yes(单分区情况则不支持)</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> programmable</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> time-based, count-based</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> yes</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important"> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Flink</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> yes</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> yes</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> programmable</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> time-based, count-based</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> yes</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important"> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Ignite Streaming</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> yes</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> yes</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> programmable</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> time-based, count-based</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> yes</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important"> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Beam</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> yes</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> yes</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> programmable</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> time-based</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> yes</td> 
 </tr> 
 </tbody> 
</table>

表一（再续）：流平台的质量属性

<table style="margin:0px 0px 10px; padding:0px; border-collapse:collapse; width:461px; max-width:100%; color:rgb(62,62,62); font-family:&quot;Hiragino Sans GB&quot;,&quot;Microsoft YaHei&quot;,Arial,sans-serif; font-size:16px; line-height:25.6px; widows:1; word-wrap:break-word!important"> 
 <thead style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important"> 
 <tr style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important"> 
 <th style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border-width:2px 1px 1px; border-style:solid; border-color:rgb(187,187,187) rgb(221,221,221) rgb(221,221,221); max-width:100%; background:rgb(247,247,247)"> 流平台</th> 
 <th style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border-width:2px 1px 1px; border-style:solid; border-color:rgb(187,187,187) rgb(221,221,221) rgb(221,221,221); max-width:100%; background:rgb(247,247,247)"> 数据抽象</th> 
 <th style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border-width:2px 1px 1px; border-style:solid; border-color:rgb(187,187,187) rgb(221,221,221) rgb(221,221,221); max-width:100%; background:rgb(247,247,247)"> 数据流</th> 
 <th style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border-width:2px 1px 1px; border-style:solid; border-color:rgb(187,187,187) rgb(221,221,221) rgb(221,221,221); max-width:100%; background:rgb(247,247,247)"> 延迟</th> 
 <th style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border-width:2px 1px 1px; border-style:solid; border-color:rgb(187,187,187) rgb(221,221,221) rgb(221,221,221); max-width:100%; background:rgb(247,247,247)"> 资源管理</th> 
 <th style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border-width:2px 1px 1px; border-style:solid; border-color:rgb(187,187,187) rgb(221,221,221) rgb(221,221,221); max-width:100%; background:rgb(247,247,247)"> Auto-scaling</th> 
 </tr> 
 </thead> 
 <tbody style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important"> 
 <tr style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important"> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Flume</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Event</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> agent</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> low</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> native</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> no</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important"> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> NiFi</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> FlowFile</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> flow</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> configurable</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> native</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> no</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important"> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Gearpump</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Message</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> streaming application</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> very low</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> YARN</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> no</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important"> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Apex</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Tuple</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> streaming application</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> very low</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> YARN</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> yes</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important"> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Kafka Streams</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> KafkaStream</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> process topology</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> very low</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> YARN, Mesos, Chef, Puppet, Salt, Kubernetes等</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> yes</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important"> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Spark Streaming</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> DStream</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> application</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> medium</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> YARN, Mesos</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> yes</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important"> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Storm</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Tuple</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> topology</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> very low</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> YARN, Mesos</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> no</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important"> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Samza</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Message</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> job</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> low</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> YARN</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> no</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important"> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Flink</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> DataStream</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> streaming dataflow</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> low(configurable)</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> YARN</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> no</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important"> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Ignite Streaming</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> IgniteDataStreamer</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> job</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> very low</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> YARN, Mesos</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> no</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important"> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Beam</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> PCollection</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> pipeline</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> low</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> integrated</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> yes</td> 
 </tr> 
 </tbody> 
</table>

表一（终）：流平台的质量属性

<table style="margin:0px 0px 10px; padding:0px; border-collapse:collapse; width:461px; max-width:100%; color:rgb(62,62,62); font-family:&quot;Hiragino Sans GB&quot;,&quot;Microsoft YaHei&quot;,Arial,sans-serif; font-size:16px; line-height:25.6px; widows:1; word-wrap:break-word!important"> 
 <thead style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important"> 
 <tr style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important"> 
 <th style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border-width:2px 1px 1px; border-style:solid; border-color:rgb(187,187,187) rgb(221,221,221) rgb(221,221,221); max-width:100%; background:rgb(247,247,247)"> 流平台</th> 
 <th style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border-width:2px 1px 1px; border-style:solid; border-color:rgb(187,187,187) rgb(221,221,221) rgb(221,221,221); max-width:100%; background:rgb(247,247,247)"> 热修改</th> 
 <th style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border-width:2px 1px 1px; border-style:solid; border-color:rgb(187,187,187) rgb(221,221,221) rgb(221,221,221); max-width:100%; background:rgb(247,247,247)"> API</th> 
 <th style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border-width:2px 1px 1px; border-style:solid; border-color:rgb(187,187,187) rgb(221,221,221) rgb(221,221,221); max-width:100%; background:rgb(247,247,247)"> 主要开发语言</th> 
 <th style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border-width:2px 1px 1px; border-style:solid; border-color:rgb(187,187,187) rgb(221,221,221) rgb(221,221,221); max-width:100%; background:rgb(247,247,247)"> API语言</th> 
 </tr> 
 </thead> 
 <tbody style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important"> 
 <tr style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important"> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Flume</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> no</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> declarative</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Java</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> text files, Java</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important"> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> NiFi</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> yes</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> compositional</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Java</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> REST(GUI)</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important"> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Gearpump</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> yes</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> declarative</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Scala</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Scala, Java</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important"> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Apex</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> yes</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> declarative</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Java</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Java</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important"> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Kafka Streams</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> yes</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> declarative</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Java</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Java</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important"> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Spark Streaming</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> no</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> declarative</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Scala</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Scala, Java, Python</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important"> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Storm</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> yes</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> compositional</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Clojure</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Scala, Java, Clojure, Python, Ruby</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important"> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Samza</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> no</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> compositional</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Scala</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Java</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important"> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Flink</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> no</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> declarative</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Java</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Java, Scala, Python</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important"> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Ignite Streaming</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> no</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> declarative</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Java</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Java, .NET, C++</td> 
 </tr> 
 <tr style="margin:0px; padding:0px; max-width:100%; word-wrap:break-word!important"> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Beam</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> no</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> declarative</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Java</td> 
 <td style="margin:0px; padding:5px 10px; word-wrap:break-word!important; word-break:break-all; border:1px solid rgb(221,221,221); max-width:100%"> Java</td> 
 </tr> 
 </tbody> 
</table>

###     ###

### 数据流模型 ###

在进行流数据处理时，必然需要消费上游的数据源，并在处理数据后输出到指定的存储，以待之后的数据分析。站在流数据的角度，无论其对数据的抽象是什么，都可以视为是对消息的生产与消费。这个过程是一个数据流（data flow），那么负责参与其中的设计元素就可以称之为是“数据流模型（Data flow model）”。

不同流处理平台的数据流模型有自己的抽象定义，也提供了内建的支持。我针对Flume、Flink、Storm、Apex以及NiFi的数据流模型作了一个简单的总结。

####     ####

#### Flume ####

Flume的数据流模型是在Agent中由Source、Channel与Sink组成。

![Center][]

内建的Source支持：

*  Avro
 *  Thrift
 *  JMS
 *  Taildir
 *  Exec
 *  Spooling Directory
 *  Twitter
 *  Kafka
 *  NetCat
 *  Sequence Generator
 *  Syslog
 *  HTTP

内建的Sink支持：

*  HDFS
 *  Hive
 *  Logger
 *  Avro
 *  Thrift
 *  IRC
 *  File Roll
 *  HBase
 *  Solr
 *  Elasticsearch
 *  Kite Dataset
 *  Kafka
 *  HTTP

Flume还支持自定义Source、Sink与Channel。

####     ####

#### Flink ####

Flink将数据流模型抽象为Connector。Connector将Source与Sink连接起来，一些特殊的connector则只有Source或Sink。Flink定义的connector包括：

*  Kafka（支持Source/Sink）
 *  Elasticsearch（仅为Sink）
 *  HDFS（仅为Sink）
 *  RabbitMQ（支持Source/Sink）
 *  Amazon Kinesis Streams（支持Source/Sink）
 *  Twitter（仅为Source）
 *  NiFi（支持Sink/Source）
 *  Cassandra（仅为Sink）
 *  Redis、Flume和ActiveMQ（仅为Sink）

Flink也支持用户自定义Connector。

####     ####

#### Storm ####

Storm对数据流模型的抽象则形象地定义为Spout和Bolt。为了支持其他数据源的读取，并将数据存储到指定位置，Storm提供了与诸多外部系统的集成，并针对这些外部系统去定义对应的Spout与Bolt。

![Center 1][]

Storm集成的外部系统包括：

*  Kafka：通过BrokerHosts的ZKHosts支持Spout
 *  HBase：提供HBaseBolt
 *  HDFS：提供HdfsBolt
 *  Hive：提供HiveBolt
 *  Solr：提供SolrUpdateBolt与对应的Mapper
 *  Canssandra：提供CassandraWriterBolt
 *  JDBC：提供JdbcInsertBolt与JdbcLookupBolt等
 *  JMS：提供JMS Spout与JMS Bolt
 *  Redis：提供RedisLookupBolt、RedisStoreBolt与RedisFilterBolt等
 *  Event Hubs：提供了Event Hubs Spout
 *  Elasticsearch：提供EsIndexBolt、EsPercolateBolt与EsLookupBolt等
 *  MQTT：MQTT主要用于物联网应用的轻量级发布/订阅协议，提供了对应的Spout
 *  MongoDB：提供了MongoInsertBolt、MongoUpdateBolt
 *  OpenTSDB
 *  Kinesis
 *  Druid
 *  Kestrel

Storm和Storm Trident都支持用户自定义Spout和Bolt。

####     ####

#### Apex ####

Apex将数据流模型称之为Operators，并将其分离出来，放到单独的Apex Malhar中。对于Source，它将其称之为Input Operators，对于Sink，则称为Output Operators，而Comput Operators则负责对流数据的处理。

![Center 2][]

Apex Malhar支持的Input/Output Operators包括：

*  文件系统：支持存储到HDFS、S3，也可以存储到NFS和本地文件系统
 *  关系型数据库：支持Oracle、MySQL、Sqlite等
 *  NoSQL数据库：支持HBase、Cassandra、Accumulo、Aerospike、MongoDB和CouchDB
 *  消息系统：支持对Kafka、JMS、ZeroMQ和RabbitMQ消息的读写
 *  通知系统：支持通过SMTP发送通知
 *  内存数据库和缓存：支持Memcached和Redis
 *  社交媒体：支持Twitter
 *  协议：支持HTTP、RSS、Socket、WebSocket、FTP和MQTT

毫无疑问，Apex也支持用户自定义Operator。除了可以用Java编写之外，还可以使用JavaScript、Python、R和Ruby。

####     ####

#### NiFi ####

NiFi对流模型的主要抽象为Processor，并且提供了非常丰富的数据源与数据目标的支持。

![Center 3][]

常用的数据采集方法包括：

*  GetFile
 *  GetFtp
 *  GetSFtp
 *  GetJMSQueue
 *  GetJMSTopic
 *  GetHTTP
 *  ListenHTTP
 *  ListenUDP
 *  GetHDFS
 *  ListHDFS / FetchHDFS
 *  FetchS3Objet
 *  GetKafka
 *  GetMongo
 *  GetTwitter

发送数据的方法包括：

*  PutEmail
 *  PutFile
 *  PutFTP
 *  putSFTP
 *  PutJMS
 *  PutSQL
 *  PutKafka
 *  PutMongo

Nifi也支持用户自定义Processor，例如通过继承NiFi定义的AbstractProcessor类。自定义的Processor可以和内建的Processor一样添加到NiFi定义Flow的GUI上，并对其进行配置。

> 出处：http://zhangyi.xyz/technical-choice-of-streaming-platform/

版权申明：内容来源网络，版权归原创者所有。除非无法确认，我们都会标明作者及出处，如有侵权烦请告知，我们会立即删除并表示歉意。谢谢。

[Center]: /images/20220529/40e65cebf0c641ccbf69232fa52c7f71.png
[Center 1]: /images/20220529/7b0960e147764a8483b27f0f62c91624.png
[Center 2]: /images/20220529/133b8f9208154f819dd9f84a54261140.png
[Center 3]: /images/20220529/34b3b5b3c58e4168ae70dae4cb65f282.png