Zeppelin client执行flink sql on hive流程备份

落日映苍穹つ 2022-08-30 15:00 238阅读 0赞

主要目的：防止自己忘记，重新记录一下  
  
1，Flink 1.11.2版本 集群配置  
查看官网地址：

[https://ci.apache.org/projects/flink/flink-docs-release-1.12/zh/dev/table/connectors/hive/][https_ci.apache.org_projects_flink_flink-docs-release-1.12_zh_dev_table_connectors_hive]

找到对应hive版本:

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMxODY2Nzkz_size_16_color_FFFFFF_t_70][]

下载这几个依赖包，放入到Flink 集群lib下:

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMxODY2Nzkz_size_16_color_FFFFFF_t_70 1][]

2, zeppelin的flink和hive配置 整合

地址：

[https://www.yuque.com/jeffzhangjianfeng/gldg8w/agf94n][https_www.yuque.com_jeffzhangjianfeng_gldg8w_agf94n]

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMxODY2Nzkz_size_16_color_FFFFFF_t_70 2][]

如下图：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMxODY2Nzkz_size_16_color_FFFFFF_t_70 3][]

3，使用Zeppelin

简单的创建一个查询，执行成功

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMxODY2Nzkz_size_16_color_FFFFFF_t_70 4][]

4，使用Zeppelin client代码执行

我们先写一个简单的paragraph：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMxODY2Nzkz_size_16_color_FFFFFF_t_70 5][]

这个任务目前是没有运行状态。

我们运行本地代码执行Zeppelin client：

public class ZeppelinClientExec_test {
    
     public static void main(String[] args) throws Exception {
       ClientConfig clientConfig = new ClientConfig("http://bi-526:18080");
       ZeppelinClient zClient = new ZeppelinClient(clientConfig);
    
       String zeppelinVersion = zClient.getVersion();
       System.out.println("Zeppelin version: " + zeppelinVersion);
    
       //todo 创建 note 包名字
    
    
       try {
    
            //todo 已经存在的noteId
           String noteId = "2GAM7NK2G";
         //todo 我们这里的 noteId =  2GDTRU4U1
    //       NoteResult noteResult = zClient.executeNote(noteId);
    //       System.out.println("Execute note and the note result: " + noteResult);
    
           String paragraphId = "paragraph_1626839477148_1814189847";
    
    
           /*
            //todo   executeParagraph 和 submitParagraph的区别是 ：
                executeParagraph 一直等待，submitParagraph不是一直等待
            */
           ParagraphResult paragraphResult = zClient.executeParagraph(noteId, paragraphId);
           List<Result> results = paragraphResult.getResults();
           System.out.println("results = " + results);
           String resultInText = paragraphResult.getResultInText();
           System.out.println("resultInText = " + resultInText);
           System.out.println("Paragraph result: " + paragraphResult);
           Status status = paragraphResult.getStatus();
           System.out.println("status = " + status);
           if ("FINISHED".equals(status)){
    
               System.out.println("任务结束.............");
           }
    
           //todo 提交之后没有返回结果
    //       ParagraphResult submitParagraph =  zClient.submitParagraph(noteId, paragraphId);
    //       zClient.waitUtilParagraphRunning(noteId, paragraphId);
    //       System.out.println("Paragraph result: " + submitParagraph);
           //todo 打印输出为：Paragraph result: ParagraphResult{paragraphId='paragraph_1626782724461_1312182058', status=PENDING, results=, progress=0}
    
       } finally {
    //     if (noteId != null) {
    //       zClient.deleteNote(noteId);
    //       System.out.println("Note " + noteId + " is deleted");
    //     }
       }
     }
    }

注意：

noteId  就是：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMxODY2Nzkz_size_16_color_FFFFFF_t_70 6][]

paragraphId就是：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMxODY2Nzkz_size_16_color_FFFFFF_t_70 7][]

执行中：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMxODY2Nzkz_size_16_color_FFFFFF_t_70 8][]

web端Zeppelin状态自动开始执行了 ：

![20210721141650842.png][]

控制台执行结果：

> Zeppelin version: 0.9.1-SNAPSHOT  
> results = \[Result\{type='TABLE', data='EXPR$0  
> 18823  
> '\}\]  
> resultInText = EXPR$0  
> 18823
> 
>   
> Paragraph result: ParagraphResult\{paragraphId='paragraph\_1626839477148\_1814189847', status=FINISHED, results=Result\{type='TABLE', data='EXPR$0  
> 18823  
> '\}, progress=0\}  
> status = FINISHED  
>

参考：

[https://www.yuque.com/jeffzhangjianfeng/gldg8w/pz2xoh][https_www.yuque.com_jeffzhangjianfeng_gldg8w_pz2xoh]

[https_ci.apache.org_projects_flink_flink-docs-release-1.12_zh_dev_table_connectors_hive]: https://ci.apache.org/projects/flink/flink-docs-release-1.12/zh/dev/table/connectors/hive/
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMxODY2Nzkz_size_16_color_FFFFFF_t_70]: /images/20220829/973f17f87b3e46fda7614dfcd2e6f78f.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMxODY2Nzkz_size_16_color_FFFFFF_t_70 1]: /images/20220829/555408f00eae4a6dabd2e85b53ca3aa1.png
[https_www.yuque.com_jeffzhangjianfeng_gldg8w_agf94n]: https://www.yuque.com/jeffzhangjianfeng/gldg8w/agf94n
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMxODY2Nzkz_size_16_color_FFFFFF_t_70 2]: /images/20220829/57f125e1e21d4411b54dc5f6c958a0d6.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMxODY2Nzkz_size_16_color_FFFFFF_t_70 3]: /images/20220829/73f58b558b50409e9f0309825c32b2c2.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMxODY2Nzkz_size_16_color_FFFFFF_t_70 4]: /images/20220829/138eb058f7264e249f10c6307e2dceb7.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMxODY2Nzkz_size_16_color_FFFFFF_t_70 5]: /images/20220829/c0a90a9a98564d63ae9eec2aef4b508b.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMxODY2Nzkz_size_16_color_FFFFFF_t_70 6]: /images/20220829/5c647991af5d45d5a9a824fc62bbfbb6.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMxODY2Nzkz_size_16_color_FFFFFF_t_70 7]: /images/20220829/9da5409a3a6d4f90a7fd8aff47c74bb8.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzMxODY2Nzkz_size_16_color_FFFFFF_t_70 8]: /images/20220829/f96d4b1d173041c09ee8cc70e7cf81c4.png
[20210721141650842.png]: /images/20220829/f9f008f671464549a08394e2e48a686f.png
[https_www.yuque.com_jeffzhangjianfeng_gldg8w_pz2xoh]: https://www.yuque.com/jeffzhangjianfeng/gldg8w/pz2xoh