flink写入hive的时区问题

r囧r小猫 2023-01-06 01:57 137阅读 0赞

### 概述 ###

本文主要对\[3\]进行复现和阐述

### **环境版本** ###

<table style="width:500px;"> 
 <tbody> 
  <tr> 
   <td style="width:254px;"><strong>组件</strong></td> 
   <td style="width:244px;"><strong>版本</strong></td> 
  </tr> 
  <tr> 
   <td style="width:254px;">Hadoop</td> 
   <td style="width:244px;">3.1.2</td> 
  </tr> 
  <tr> 
   <td style="width:254px;">Hive</td> 
   <td style="width:244px;">2.3.6</td> 
  </tr> 
  <tr> 
   <td style="width:254px;">Flink</td> 
   <td style="width:244px;">1.12.0</td> 
  </tr> 
  <tr> 
   <td style="width:254px;">Zookeeper</td> 
   <td style="width:244px;">3.6.0</td> 
  </tr> 
 </tbody> 
</table>

\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#\#

### **本文流水线触发条件** ###

我们目前接触到的有下面两种:

<table style="width:586px;"> 
 <tbody> 
  <tr> 
   <td style="width:203px;"><strong>环境</strong></td> 
   <td style="width:381px;"><strong>触发计算条件(为了方便论述，这里忽略迟到数据等)</strong></td> 
  </tr> 
  <tr> 
   <td style="width:203px;"> <p>watermark写入hive分区</p> </td> 
   <td style="width:381px;">watermark&nbsp;&gt;&nbsp;toMills(partTime)&nbsp;+&nbsp;commitDelay</td> 
  </tr> 
  <tr> 
   <td style="width:203px;">watermark与window计算[4]</td> 
   <td style="width:381px;"><strong>watermark &gt;= window_end_time</strong>&nbsp;</td> 
  </tr> 
 </tbody> 
</table>

**本文论述的是第一种**

百度上常见的是表格中第二种

### 时区机制 ###

Flink源码中的toMills并不是jdk中的

java.time.Duration.toMillis()

两者不是一个东西，不要搞混．

上述表格中的toMills会把传入的创建hive分区的时间戳partTime误当做UTC时间(作为输入)，

然后+8小时转化为东八区的时间toMills(partTime)．

整理如下:

<table style="width:592px;"> 
 <tbody> 
  <tr> 
   <td>&nbsp;</td> 
   <td style="width:134px;"><strong>使用的数据来自</strong></td> 
   <td style="width:123px;"><strong>被Flink误当做</strong></td> 
   <td style="width:173px;"><strong>进一步转化为</strong></td> 
  </tr> 
  <tr> 
   <td>watermark</td> 
   <td style="width:134px;">东八区</td> 
   <td style="width:123px;">UTC时区</td> 
   <td style="width:173px;">无</td> 
  </tr> 
  <tr> 
   <td>partTime(创建hive分区的时间)</td> 
   <td style="width:134px;">东八区</td> 
   <td style="width:123px;">UTC时区</td> 
   <td style="width:173px;"> <p>toMills(partTime)=partTime+8:00</p> </td> 
  </tr> 
 </tbody> 
</table>

**所以这就尴尬了，时区失配，导致无案发触发Partition Commit(讲人话就是Flink已经写入Sink的数据在Hive中看不到)．**  
场景举例:

16:23:48我新建了一个hive**表(当然也就等于是新建了hive分区)**

**16:26:25时我去hive表格中查看数据，结果一条都没有．**

**为啥呢？**

**根据上述表格，我建立hive中表格fs\_table的时间，被flink加了8小时，需要当前写入的数据的时间戳生成的watermark**

**大于0:23:48(**16:23:48+8:00**)以后才能看到**

**因为没有满足表格中的水位线触发条件，所以无法触发Hive的Partition Commit，**

**最终导致一个奇怪的现象:**

**Flink明明已经写入Sink的数据,在Hive的fs\_table中看不到**

### 实验验证 ###

口说无凭，我们来做一个对照实验

<table style="width:588px;"> 
 <tbody> 
  <tr> 
   <td style="width:49px;"><strong>步骤</strong></td> 
   <td style="width:188px;"><strong>实验组</strong></td> 
   <td style="width:213px;"><strong>对照组</strong></td> 
   <td style="width:134px;"><strong>备注</strong></td> 
  </tr> 
  <tr> 
   <td style="width:49px;">①</td> 
   <td style="width:188px;"> <p><a href="https://gitee.com/appleyuchi/Flink_Code/blob/master/Flink%E6%97%B6%E5%8C%BA%E9%97%AE%E9%A2%98/sql%E5%86%85%E5%AE%B9.sql" rel="nofollow">创建Hive表格fs_table</a></p> </td> 
   <td style="width:213px;"><a href="https://gitee.com/appleyuchi/Flink_Code/blob/master/Flink%E6%97%B6%E5%8C%BA%E9%97%AE%E9%A2%98/sql%E5%86%85%E5%AE%B9.sql" rel="nofollow">创建Hive表格fs_table</a></td> 
   <td style="width:134px;">&nbsp;</td> 
  </tr> 
  <tr> 
   <td style="width:49px;">②</td> 
   <td style="width:188px;">运行<a href="https://gitee.com/appleyuchi/Flink_Code/blob/master/Flink%E6%97%B6%E5%8C%BA%E9%97%AE%E9%A2%98/src/main/java/StreamingWriteHive_bug.java" rel="nofollow">StreamingWriteHive_bug.java</a></td> 
   <td style="width:213px;">运行<a href="https://gitee.com/appleyuchi/Flink_Code/blob/master/Flink%E6%97%B6%E5%8C%BA%E9%97%AE%E9%A2%98/src/main/java/StreamingWriteHive.java" rel="nofollow">StreamingWriteHive.java</a></td> 
   <td style="width:134px;">对照组的水印加了8小时</td> 
  </tr> 
  <tr> 
   <td style="width:49px;">③</td> 
   <td style="width:188px;"> <p>select * from fs_table;</p> </td> 
   <td style="width:213px;">select * from fs_table;</td> 
   <td style="width:134px;">&nbsp;</td> 
  </tr> 
  <tr> 
   <td style="width:49px;">④</td> 
   <td style="width:188px;">Hive的fs_table中无结果</td> 
   <td style="width:213px;">Hive的fs_table中一大堆结果</td> 
   <td style="width:134px;">&nbsp;</td> 
  </tr> 
 </tbody> 
</table>

**Reference:**

\[1\][Hive分区和分桶区别][Hive]

\[2\][重要|flink的时间及时区问题解决][flink]

\[3\][flink教程-flink 1.11 使用sql将流式数据写入hive][flink_-flink 1.11 _sql_hive]

\[4\][Flink EventTime和Watermarks原理结合代码分析(转载+解决+精简记录)][Flink EventTime_Watermarks]

\[5\][踩坑记 | Flink 天级别窗口中存在的时区问题][_ Flink]

\[6\][Hive创建二级分区表][Hive 1]

[Hive]: https://blog.csdn.net/weixin_39393048/article/details/82530254
[flink]: https://blog.csdn.net/rlnlo2pnefx9c/article/details/91915550
[flink_-flink 1.11 _sql_hive]: https://my.oschina.net/u/4596020/blog/4437995
[Flink EventTime_Watermarks]: https://yuchi.blog.csdn.net/article/details/108451350
[_ Flink]: https://www.jianshu.com/p/4593c65ea5b7
[Hive 1]: https://blog.csdn.net/kwu_ganymede/article/details/49126389