自定义CDH parcel

ゝ一纸荒年。 2022-10-17 05:54 7阅读 0赞

### 文章目录 ###

*  1, 概念介绍: parcel,csd
 *  2, 自定义parcel: 把jar包封装为parcel
 *   *  a, 生成parcel
     *  b, 生成csd jar包
     *  c, 部署服务到cloudera-manager

# 1, 概念介绍: parcel,csd #

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2V5ZW9mZWFnbGU_size_16_color_FFFFFF_t_70]

<table> 
 <thead> 
  <tr> 
   <th align="left">名称</th> 
   <th>含义</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td align="left">parcel</td> 
   <td>普通tarball + metadata ( xx_env.sh ，parcel.json, alternatives.json )</td> 
  </tr> 
  <tr> 
   <td align="left">csd (Custom Service Descriptors )</td> 
   <td>普通jar包 ( descriptor/service.sdl, images/xx.png, scripts/control.sh )</td> 
  </tr> 
 </tbody> 
</table>

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2V5ZW9mZWFnbGU_size_16_color_FFFFFF_t_70 1]  
![在这里插入图片描述][2021060209314149.png]

*  parcel 详细说明：[ https://github.com/cloudera/cm\_ext/wiki/The-parcel.json-file][https_github.com_cloudera_cm_ext_wiki_The-parcel.json-file]
 *  csd 详细说明：[https://github.com/cloudera/cm\_ext/wiki/CSD-Overview][https_github.com_cloudera_cm_ext_wiki_CSD-Overview]， [https://github.com/cloudera/cm\_ext/wiki/Service-Descriptor-Language-Reference][https_github.com_cloudera_cm_ext_wiki_Service-Descriptor-Language-Reference]

# 2, 自定义parcel: 把jar包封装为parcel #

## a, 生成parcel ##

*  获取parcel制作/验证的工具：[https://github.com/cloudera/cm\_ext/][https_github.com_cloudera_cm_ext]  
    [百度盘-提取码：abcd][-_abcd]

[root@localhost Downloads]# tree DEMO-2.0/
    DEMO-2.0/
    ├── lib
    │   └── demo-2.0.jar
    └── meta
        ├── demo_env.sh
        └── parcel.json
    2 directories, 3 files
    
    [root@localhost Downloads]# java -jar DEMO-2.0/lib/demo-2.0.jar
     .   ____          _            __ _ _
     /\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \ ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
     \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
      '  |____| .__|_| |_|_| |_\__, | / / / /
     =========|_|==============|___/=/_/_/_/
     :: Spring Boot ::                (v2.5.0)
    2021-06-02 09:33:04.650  INFO 9524 --- [           main] com.example.demo3.Demo3Application       
    ^C[root@localhost Downloads]#
    [root@localhost Downloads]# cd DEMO-2.0/
    
    [root@localhost DEMO-2.0]# cat meta/demo_env.sh
    #!/bin/bash
    DEMO_DIRNAME=${ PARCEL_DIRNAME:-"DEMO-2.0"}
    export DEMO_HOME=$PARCELS_ROOT/$DEMO_DIRNAME/lib
    
    [root@localhost DEMO-2.0]# cat meta/parcel.json
    { 
        "schema_version":1,
        "name":"DEMO",
        "version":"2.0",
        "setActiveSymlink":true,
        "conflicts":"",
        "provides":[
            "demo"
        ],
        "scripts":{ 
            "defines":"demo_env.sh"
        },
        "packages":[
            { 
                "name":"demo",
                "version":"2.0"
            }
        ],
        "components":[
            { 
                "name":"demo",
                "version":"2.0",
                "pkg_version":"2.0",
                "pkg_release":"2.0"
            }
        ],
        "users":{  },
        "groups":[  ]
    }
    
    [root@localhost DEMO-2.0]# cd ..
    [root@localhost Downloads]# mkdir parcels
    [root@localhost Downloads]# tar -zcvf DEMO-2.0-el6.parcel DEMO-2.0/
    DEMO-2.0/
    DEMO-2.0/lib/
    DEMO-2.0/lib/demo-2.0.jar
    DEMO-2.0/meta/
    DEMO-2.0/meta/parcel.json
    DEMO-2.0/meta/demo_env.sh
    [root@localhost Downloads]# mv DEMO-2.0-el6.parcel parcels/
    [root@localhost Downloads]# cd parcels/
    [root@localhost parcels]# ls
    DEMO-2.0-el6.parcel    make_manifest.py  validator.jar
    [root@localhost parcels]# java -jar validator.jar -f DEMO-2.0-el6.parcel
    Validating: DEMO-2.0-el6.parcel
    Validating: DEMO-2.0/meta/parcel.json
    Validation succeeded.
    
    [root@localhost parcels]# python make_manifest.py
    Scanning directory: .
    Found parcel DEMO-2.0-el6.parcel
    [root@localhost parcels]# ls
    DEMO-2.0-el6.parcel   make_manifest.py  manifest.json  validator.jar
    [root@localhost parcels]# cat manifest.json
    { 
        "parcels": [
            { 
                "conflicts": "",
                "hash": "faac5c55c6e918f4f77bac572ba69ae577386750",
                "parcelName": "DEMO-2.0-el6.parcel",
                "components": [
                    { 
                        "pkg_version": "2.0",
                        "version": "2.0",
                        "name": "demo",
                        "pkg_release": "2.0"
                    }
                ]
            }
        ],
        "lastUpdated": 1622598123811
    }[root@localhost parcels]# cat manifest.json | grep hash | awk -F"\"" '{print$4}' > DEMO-2.0-el6.parcel.sha
    [root@localhost parcels]# cat DEMO-2.0-el6.parcel.sha
    faac5c55c6e918f4f77bac572ba69ae577386750
    [root@localhost parcels]# sha1sum DEMO-2.0-el6.parcel
    faac5c55c6e918f4f77bac572ba69ae577386750  DEMO-2.0-el6.parcel

## b, 生成csd jar包 ##

*  csd 配置模板(kafka, ECHO, spark , sqoop ) [https://github.com/cloudera/cm\_csds][https_github.com_cloudera_cm_csds]
 *  csd jar包配置样例 （ECHO service )：[https://github.com/cloudera/cm\_ext/wiki/CSD-Primer][https_github.com_cloudera_cm_ext_wiki_CSD-Primer]
 *  png图片大小: 16\*16 最好，否则页面显示有问题  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2V5ZW9mZWFnbGU_size_16_color_FFFFFF_t_70 2]  
    service.sdl

{ 
        "name":"DEMO",
        "label":"Demo",
        "description":"spring boot demo application",
        "version":"1.0",
        "runAs":{ 
            "user":"root",
            "group":"root"
        },
        "parcel":{ 
            "requiredTags":[
                "demo"
            ]
        },
        "compatibility":{ 
            "generation":1
        },
        "icon":"images/binocular.png",
        "parameters":[
            { 
                "name":"service_usage",
                "label":"Service usage for ",
                "description":"what is the tomcat used for, ie:测试,运行服务1...",
                "configName":"service.usage",
                "type":"string",
                "default":"default tomcat service",
                "configurableInWizard":true
            },
            { 
                "name":"start_port",
                "label":"start port: listener port",
                "description":"服务访问/监听端口",
                "required":"true",
                "type":"long",
                "default":8810,
                "min":8000,
                "configurableInWizard":true
            },
            { 
                "name":"stop_port",
                "label":"shutdown port",
                "description":"服务关闭端口",
                "required":"true",
                "type":"long",
                "default":8005,
                "min":8000,
                "configurableInWizard":true
            }
        ],
        "roles":[
            { 
                "name":"DEMO_WEBSERVER",
                "label":"Web Server",
                "pluralLabel":"Web Servers",
                "startRunner":{ 
                    "program":"scripts/control.sh",
                    "args":[
                        "start"
                    ],
                    "environmentVariables":{ 
                        "stop_port":"${stop_port}",
                        "start_port":"${start_port}",
                        "service_usage":"${service_usage}"
                    }
                },
                "logging":{ 
                    "dir":"/var/log/demo",
                    "filename":"demo-${start_port}.log",
                    "configName":"log.dir",
                    "loggingType":"other"
                }
            }
        ]
    }

control.sh

#!/bin/bash
    # Time marker for both stderr and stdout
    date; date 1>&2
    
    cmd=$1
    timestamp=$(date)
    export DEMO_HOME=$PARCELS_ROOT/$DEMO_DIRNAME/lib
    echo "service_usage:"$service_usage
    echo "stop_port:"$stop_port
    echo "start_port:"$start_port
    
    if [ "start" = "$cmd" ]; then
      echo "$timestamp Starting Server" 
      exec java -Dserver.port=$start_port -jar $DEMO_HOME/*.jar
    
    elif [ "stop" = "$cmd" ]; then
      echo "$timestamp Stopping Server"
    else
      echo "$timestamp Don't understand [$cmd]"
    fi

## c, 部署服务到cloudera-manager ##

发布parcel和csd包, 并重启cloudera-manager 服务 （7180所在的主机)

[root@localhost ~]# ls /opt/cloudera/parcel-repo/DEMO-2.0-el6.parcel* -l
    -rw-rw-r-- 1 root  root  15511313 Jun  2 08:57 /opt/cloudera/parcel-repo/DEMO-2.0-el6.parcel
    -rw-rw-r-- 1 root  root        41 Jun  2 08:57 /opt/cloudera/parcel-repo/DEMO-2.0-el6.parcel.sha
    
    [root@localhost ~]# ls /opt/cloudera/csd/DEMO-2.0.jar
    /opt/cloudera/csd/DEMO-2.0.jar

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2V5ZW9mZWFnbGU_size_16_color_FFFFFF_t_70 3]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2V5ZW9mZWFnbGU_size_16_color_FFFFFF_t_70 4]

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2V5ZW9mZWFnbGU_size_16_color_FFFFFF_t_70 5]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2V5ZW9mZWFnbGU_size_16_color_FFFFFF_t_70 6]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2V5ZW9mZWFnbGU_size_16_color_FFFFFF_t_70 7]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2V5ZW9mZWFnbGU_size_16_color_FFFFFF_t_70]: /images/20221014/7206d210baa54f3ebb019ea76587001b.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2V5ZW9mZWFnbGU_size_16_color_FFFFFF_t_70 1]: /images/20221014/60a39f06530643cda3eca50c49bbc0d9.png
[2021060209314149.png]: /images/20221014/1ccfb0d1fb2443d28b4c88d3d908e341.png
[https_github.com_cloudera_cm_ext_wiki_The-parcel.json-file]: https://github.com/cloudera/cm_ext/wiki/The-parcel.json-file
[https_github.com_cloudera_cm_ext_wiki_CSD-Overview]: https://github.com/cloudera/cm_ext/wiki/CSD-Overview
[https_github.com_cloudera_cm_ext_wiki_Service-Descriptor-Language-Reference]: https://github.com/cloudera/cm_ext/wiki/Service-Descriptor-Language-Reference
[https_github.com_cloudera_cm_ext]: https://github.com/cloudera/cm_ext/
[-_abcd]: https://pan.baidu.com/s/1RI-6iSL2G8AxPhbSkan68A
[https_github.com_cloudera_cm_csds]: https://github.com/cloudera/cm_csds
[https_github.com_cloudera_cm_ext_wiki_CSD-Primer]: https://github.com/cloudera/cm_ext/wiki/CSD-Primer
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2V5ZW9mZWFnbGU_size_16_color_FFFFFF_t_70 2]: /images/20221014/e92a10fbaaf34ff4b1eac37384ec6199.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2V5ZW9mZWFnbGU_size_16_color_FFFFFF_t_70 3]: /images/20221014/1a6f7312a97449008daa4d42e32c251a.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2V5ZW9mZWFnbGU_size_16_color_FFFFFF_t_70 4]: /images/20221014/3517f341d54e485e9514be154ceeead4.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2V5ZW9mZWFnbGU_size_16_color_FFFFFF_t_70 5]: /images/20221014/32fbe8e7095442f894ce800d374bee12.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2V5ZW9mZWFnbGU_size_16_color_FFFFFF_t_70 6]: /images/20221014/bf973f9f91334b48be3f3eb2909a5c6d.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2V5ZW9mZWFnbGU_size_16_color_FFFFFF_t_70 7]: /images/20221014/5d6c5a5669734879ae8775ac5a47f57d.png