Cloudera Manager安装streamsets

ゞ浴缸里的玫瑰 2022-12-14 08:52 249阅读 0赞

### 文章目录 ###

*  1, 下载streamsets的parcel文件
 *  2, 使用cloudera manager安装parcel
 *   *  a, 分发parcel报错
     *   *  报错后，下载好的parcel被删除
     *  b, 错误解决

![在这里插入图片描述][20201014150245430.png_pic_center]

# 1, 下载streamsets的parcel文件 #

parcel下载地址： [https://archives.streamsets.com/datacollector][https_archives.streamsets.com_datacollector]  
parcel使用cloudera manager安装文档：[https://streamsets.com/documentation/datacollector/3.16.x/help/datacollector/UserGuide/Installation/CMInstall-Overview.html\#concept\_nb5\_c3m\_25][https_streamsets.com_documentation_datacollector_3.16.x_help_datacollector_UserGuide_Installation_CMInstall-Overview.html_concept_nb5_c3m_25]

*  Install the StreamSets custom service descriptor (CSD). (需要重启cm\_server, cm监控服务)
 *  (Optional.) Manually install the parcel and checksum files. Typically only needed when the Cloudera Manager Server does not have internet access.
 *  Download, distribute, and activate the StreamSets parcel.

# 2, 使用cloudera manager安装parcel #

## a, 分发parcel报错 ##

*  问题1: host\_monitor退出,
 *  问题2: parcel下载后解压失败被删除
 *  问题3: 分发parcel卡住

=================第一次：agent, 下载解压报错
    [root@test-c6 ~]# find /opt/ -name "STREAM*"
    /opt/cloudera/parcel-repo/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel
    /opt/cloudera/parcel-repo/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel.torrent
    /opt/cloudera/parcel-repo/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel.sha
    /opt/cloudera/csd/STREAMSETS-3.16.2.jar
    /opt/cloudera/parcel-cache/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel.torrent
    /opt/cloudera/parcels/.flood/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel
    /opt/cloudera/parcels/.flood/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel
    /opt/cloudera/parcels/.flood/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel.torrent
    /opt/cloudera/parcels/STREAMSETS_DATACOLLECTOR
    /opt/cloudera/parcels/STREAMSETS_DATACOLLECTOR-3.16.2
    
    [root@test-c6 parcel-repo]# tailf /opt/cm-5.12.2/log/cloudera-scm-agent/cloudera-scm-agent.log
    [11/Oct/2020 01:36:34 +0000] 28799 Thread-13 downloader   INFO     Current state: STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel [totalDownloaded=5611142ize=5712330086 upload=0 state=downloading seed=['http://test-c6:7180/cmf/parcel/download/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel'] location=/opt/crcels/.flood/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel progress=982286]
    [11/Oct/2020 01:36:36 +0000] 28799 Thread-13 downloader   INFO     Current state: STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel [totalDownloaded=5712330ize=5712330086 upload=0 state=seeding seed=['http://test-c6:7180/cmf/parcel/download/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel'] location=/opt/clouds/.flood/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel progress=1000000]
    [11/Oct/2020 01:36:36 +0000] 28799 Thread-13 downloader   INFO     Completed download of http://test-c6:7180/cmf/parcel/download/STREAMSETS_DATACOLLE2-el6.parcel code=200 state=downloaded
    [11/Oct/2020 01:36:36 +0000] 28799 Thread-13 parcel_cache WARNING  No checksum in header, skipping verification
    [11/Oct/2020 01:36:36 +0000] 28799 Thread-13 parcel_cache INFO     Unpacking /opt/cloudera/parcels/.flood/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel/_DATACOLLECTOR-3.16.2-el6.parcel into /opt/cloudera/parcels
    [11/Oct/2020 01:37:08 +0000] 28799 MainThread agent        INFO     PID '30453' associated with process '124-cloudera-mgmt-SERVICEMONITOR' with paylosname:124-cloudera-mgmt-SERVICEMONITOR groupname:124-cloudera-mgmt-SERVICEMONITOR from_state:RUNNING expected:0 pid:30453' exited unexpectedly [11/Oct/2020 01:37:34 +0000] 28799 Monitor-GenericMonitor throttling_logger ERROR Error fetching metrics at 'http://test-c6.com:8091/jmx' Traceback (most recent call last): File "/opt/cm-5.12.2/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.12.2-py2.6.egg/cmf/monitor/generic/metric_collectors.py", line 200ect_and_parse_and_return self._adapter.safety_valve)) File "/opt/cm-5.12.2/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.12.2-py2.6.egg/cmf/url_util.py", line 207, in urlopen_with_retry_ocation_errors return function() File "/opt/cm-5.12.2/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.12.2-py2.6.egg/cmf/monitor/generic/metric_collectors.py", line 217_url password=self._password_value) File "/opt/cm-5.12.2/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.12.2-py2.6.egg/cmf/url_util.py", line 70, in urlopen_with_timeout return opener.open(url, data, timeout) File "/usr/lib64/python2.6/urllib2.py", line 391, in open response = self._open(req, data) File "/usr/lib64/python2.6/urllib2.py", line 409, in _open '_open', req) File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain result = func(*args) File "/usr/lib64/python2.6/urllib2.py", line 1190, in http_open return self.do_open(httplib.HTTPConnection, req) File "/usr/lib64/python2.6/urllib2.py", line 1165, in do_open raise URLError(err) URLError: <urlopen error [Errno 104] Connection reset by peer> [11/Oct/2020 01:37:35 +0000] 28799 MonitorDaemon-Reporter proc_metrics_utils ERROR Failed to read file descriptor max for process 30381: [Errno 2]ile or directory: '/proc/30381/limits' [11/Oct/2020 01:37:35 +0000] 28799 MonitorDaemon-Reporter proc_metrics_utils ERROR Failed to get file descriptor count for process 30381: [Errno 2file or directory: '/proc/30381/fd/' [11/Oct/2020 01:37:35 +0000] 28799 MonitorDaemon-Reporter proc_metrics_utils ERROR Failed to get process metrics 30381: no process found with pid [11/Oct/2020 01:37:36 +0000] 28799 MonitorDaemon-Reporter throttling_logger ERROR Error sending messages to firehose: mgmt-SERVICEMONITOR-af401409538410569c49e2 Traceback (most recent call last): File "/opt/cm-5.12.2/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.12.2-py2.6.egg/cmf/monitor/firehose.py", line 120, in _send self._requestor.request('sendAgentMessages', dict(messages=messages)) File "/opt/cm-5.12.2/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py", line 141, in request return self.issue_request(call_request, message_name, request_datum) File "/opt/cm-5.12.2/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py", line 254, in issue_request call_response = self.transceiver.transceive(call_request) File "/opt/cm-5.12.2/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py", line 482, in transceive self.write_framed_message(request) File "/opt/cm-5.12.2/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py", line 501, in write_framed_message self.conn.request(req_method, self.req_resource, req_body, req_headers) File "/usr/lib64/python2.6/httplib.py", line 979, in request self._send_request(method, url, body, headers) File "/usr/lib64/python2.6/httplib.py", line 1010, in _send_request self.endheaders() File "/usr/lib64/python2.6/httplib.py", line 967, in endheaders self._send_output() File "/usr/lib64/python2.6/httplib.py", line 831, in _send_output self.send(msg) File "/usr/lib64/python2.6/httplib.py", line 790, in send self.connect() File "/usr/lib64/python2.6/httplib.py", line 771, in connect self.timeout) File "/usr/lib64/python2.6/socket.py", line 567, in create_connection raise error, msg error: [Errno 111] Connection refused [11/Oct/2020 01:37:36 +0000] 28799 MonitorDaemon-Reporter throttling_logger ERROR Error sending messages to firehose: mgmt-HOSTMONITOR-af401409b8a410569c49e2 Traceback (most recent call last): File "/opt/cm-5.12.2/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.12.2-py2.6.egg/cmf/monitor/firehose.py", line 120, in _send self._requestor.request('sendAgentMessages', dict(messages=messages)) File "/opt/cm-5.12.2/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py", line 141, in request return self.issue_request(call_request, message_name, request_datum) File "/opt/cm-5.12.2/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py", line 254, in issue_request call_response = self.transceiver.transceive(call_request) File "/opt/cm-5.12.2/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py", line 482, in transceive self.write_framed_message(request) File "/opt/cm-5.12.2/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py", line 501, in write_framed_message self.conn.request(req_method, self.req_resource, req_body, req_headers) File "/usr/lib64/python2.6/httplib.py", line 979, in request self._send_request(method, url, body, headers) File "/usr/lib64/python2.6/httplib.py", line 1010, in _send_request self.endheaders() File "/usr/lib64/python2.6/httplib.py", line 967, in endheaders self._send_output() File "/usr/lib64/python2.6/httplib.py", line 831, in _send_output self.send(msg) File "/usr/lib64/python2.6/httplib.py", line 790, in send self.connect() File "/usr/lib64/python2.6/httplib.py", line 771, in connect self.timeout) File "/usr/lib64/python2.6/socket.py", line 567, in create_connection raise error, msg error: [Errno 111] Connection refused [11/Oct/2020 01:37:36 +0000] 28799 MainThread agent INFO PID '30381' associated with process '121-cloudera-mgmt-HOSTMONITOR' with payload me:121-cloudera-mgmt-HOSTMONITOR groupname:121-cloudera-mgmt-HOSTMONITOR from_state:RUNNING expected:0 pid:30381' exited unexpectedly
    [11/Oct/2020 01:38:03 +0000] 28799 MainThread agent        INFO     PID '30243' associated with process '121-cloudera-mgmt-HOSTMONITOR' with payload me:121-cloudera-mgmt-HOSTMONITOR groupname:121-cloudera-mgmt-HOSTMONITOR from_state:RUNNING expected:0 pid:30243' exited unexpectedly [11/Oct/2020 01:38:07 +0000] 28799 MainThread agent INFO PID '30316' associated with process '124-cloudera-mgmt-SERVICEMONITOR' with paylosname:124-cloudera-mgmt-SERVICEMONITOR groupname:124-cloudera-mgmt-SERVICEMONITOR from_state:RUNNING expected:0 pid:30316' exited unexpectedly
    [11/Oct/2020 01:38:28 +0000] 28799 MainThread agent        INFO     PID '30416' associated with process '121-cloudera-mgmt-HOSTMONITOR' with payload me:121-cloudera-mgmt-HOSTMONITOR groupname:121-cloudera-mgmt-HOSTMONITOR from_state:RUNNING expected:0 pid:30416' exited unexpectedly

### 报错后，下载好的parcel被删除 ###

#=================之后, /opt/cloudera/parcels/.flood/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel文件被删除, 并开始报下面的错
    [root@test-c6 parcel-repo]# find /opt/ -name "STREAM*"
    /opt/cloudera/parcel-repo/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel
    /opt/cloudera/parcel-repo/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel.torrent
    /opt/cloudera/parcel-repo/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel.sha
    /opt/cloudera/csd/STREAMSETS-3.16.2.jar
    /opt/cloudera/parcel-cache/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel.torrent
    /opt/cloudera/parcels/.flood/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel
    /opt/cloudera/parcels/.flood/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel.torrent
    
    [root@test-c6 parcel-repo]# tailf /opt/cm-5.12.2/log/cloudera-scm-agent/cloudera-scm-agent.log
    [11/Oct/2020 01:59:47 +0000] 1594 Thread-13 downloader   INFO     Fetching torrent: http://test-c6:7180/cmf/parcel/download/STREAMSETS_DATACOLLECTOR-.parcel.torrent
    [11/Oct/2020 01:59:47 +0000] 1594 Thread-13 downloader   INFO     Starting download of: http://test-c6:7180/cmf/parcel/download/STREAMSETS_DATACOLLEC-el6.parcel
    [11/Oct/2020 01:59:47 +0000] 1594 Thread-13 downloader   INFO     Failed adding torrent: file:///opt/cloudera/parcel-cache/STREAMSETS_DATACOLLECTOR-3parcel.torrent Already present torrent: STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel
    [11/Oct/2020 01:59:47 +0000] 1594 Thread-13 downloader   INFO     Current state: STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel [totalDownloaded=57123300ze=5712330086 upload=0 state=seeding seed=['http://test-c6:7180/cmf/parcel/download/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel'] location=/opt/cloude/.flood/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel progress=1000000]
    [11/Oct/2020 01:59:47 +0000] 1594 Thread-13 downloader   INFO     Completed download of http://test-c6:7180/cmf/parcel/download/STREAMSETS_DATACOLLEC-el6.parcel code=200 state=downloaded
    [11/Oct/2020 01:59:47 +0000] 1594 Thread-13 parcel_cache WARNING  No checksum in header, skipping verification
    [11/Oct/2020 01:59:47 +0000] 1594 Thread-13 parcel_cache INFO     Unpacking /opt/cloudera/parcels/.flood/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel/SDATACOLLECTOR-3.16.2-el6.parcel into /opt/cloudera/parcels
    [11/Oct/2020 01:59:47 +0000] 1594 Thread-13 downloader   ERROR    Failed op: Src file /opt/cloudera/parcels/.flood/STREAMSETS_DATACOLLECTOR-3.16.2-elTREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel does not exist
    Traceback (most recent call last):
      File "/opt/cm-5.12.2/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.12.2-py2.6.egg/cmf/downloader.py", line 501, in callable
        callback(url, curr_op)
      File "/opt/cm-5.12.2/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.12.2-py2.6.egg/cmf/parcel_cache.py", line 203, in cb
        raise e
    Exception: Src file /opt/cloudera/parcels/.flood/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel does not exist
    [11/Oct/2020 01:59:47 +0000] 1594 Thread-13 downloader   INFO     Finished download [ url: http://test-c6:7180/cmf/parcel/download/STREAMSETS_DATACOL6.2-el6.parcel, state: exception, total_bytes: 5712330086, downloaded_bytes: 5712330086, start_time: 2020-10-11 01:59:47, download_end_time: 2020-10-7, end_time: 2020-10-11 01:59:47, code: 601, exception_msg: Src file /opt/cloudera/parcels/.flood/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel/STREAMSELECTOR-3.16.2-el6.parcel does not exist, path: /opt/cloudera/parcels/.flood/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel/STREAMSETS_DATACOLLECTOR-3.16.el ]

## b, 错误解决 ##

*  第一步: host\_monitor,service\_monitor 配置修改: jvm内存=1g, 文件描述符数=65536
 *  第二步: 系统参数修改 /etc/security/limits.d/的nofile值=65536, 重启sshd
 *  第三步: 重启监控服务, 重启agent （触发再次下载parcel)

#=================第二次：agent重启(修改host_monitor,service_monitor 的jvm参数1g,
    				    Cloudera Management Service的最大进程文件描述符数=65536, 
    				    和/etc/security/limits.d/的nofile值=65536, 重启sshd, 
    				    重启监控服务, 重启agent) 
    			下载解压正常
    			
    [11/Oct/2020 04:20:03 +0000] 18179 Thread-13 downloader   INFO     Current state: STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel [totalDownloaded=56169095712330086 upload=0 state=downloading seed=['http://test-c6:7180/cmf/parcel/download/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel'] location=/opt/cloudlood/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel progress=983295]
    [11/Oct/2020 04:20:05 +0000] 18179 Thread-13 downloader   INFO     Current state: STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel [totalDownloaded=57123305712330086 upload=0 state=seeding seed=['http://test-c6:7180/cmf/parcel/download/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel'] location=/opt/cloudera//STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel progress=1000000]
    [11/Oct/2020 04:20:05 +0000] 18179 Thread-13 downloader   INFO     Completed download of http://test-c6:7180/cmf/parcel/download/STREAMSETS_DATACOLLE6.parcel code=200 state=downloaded
    [11/Oct/2020 04:20:05 +0000] 18179 Thread-13 parcel_cache WARNING  No checksum in header, skipping verification
    [11/Oct/2020 04:20:05 +0000] 18179 Thread-13 parcel_cache INFO     Unpacking /opt/cloudera/parcels/.flood/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel/ACOLLECTOR-3.16.2-el6.parcel into /opt/cloudera/parcels
    [11/Oct/2020 04:21:43 +0000] 18179 MainThread agent        WARNING  Long HB processing time: 6.20692515373
    
    [11/Oct/2020 04:22:20 +0000] 18179 Thread-13 parcel_cache INFO     Unpack of parcel /opt/cloudera/parcels/.flood/STREAMSETS_DATACOLLECTOR-3.16.2-el6.ETS_DATACOLLECTOR-3.16.2-el6.parcel successful
    [11/Oct/2020 04:22:20 +0000] 18179 Thread-13 downloader   INFO     Finished download [ url: http://test-c6:7180/cmf/parcel/download/STREAMSETS_DATACO-el6.parcel, state: complete, total_bytes: 5712330086, downloaded_bytes: 5712330086, start_time: 2020-10-11 04:15:14, download_end_time: 2020-10-11 0ime: 2020-10-11 04:22:20, code: 200, exception_msg: None, path: /opt/cloudera/parcels/.flood/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel/STREAMSETS_DA16.2-el6.parcel ]
    [11/Oct/2020 04:22:22 +0000] 18179 MainThread parcel       INFO     Loading parcel manifest for: CDH-5.12.0-1.cdh5.12.0.p0.29
    [11/Oct/2020 04:22:22 +0000] 18179 MainThread parcel       INFO     Loading parcel manifest for: STREAMSETS_DATACOLLECTOR-3.16.2
    [11/Oct/2020 04:22:22 +0000] 18179 MainThread parcel       INFO     Ensuring users/groups exist for new parcel STREAMSETS_DATACOLLECTOR-3.16.2.
    [11/Oct/2020 04:22:22 +0000] 18179 MainThread parcel       INFO     Executing command ['/usr/sbin/groupadd', '-r', 'sdc']
    [11/Oct/2020 04:22:22 +0000] 18179 MainThread parcel       INFO     Executing command ['/usr/sbin/groupadd', '-r', 'sdc']
    [11/Oct/2020 04:22:22 +0000] 18179 MainThread parcel       INFO     Executing command ['/usr/sbin/useradd', '-r', '-m', '-g', 'sdc', '-K', 'UMASK=022/var/lib/sdc', '--comment', 'StreamSets Data Collector', '--shell', '/sbin/nologin', 'sdc']
    [11/Oct/2020 04:22:22 +0000] 18179 MainThread parcel       INFO     Ensuring correct file permissions for new parcel STREAMSETS_DATACOLLECTOR-3.16.2.
    [11/Oct/2020 04:25:22 +0000] 18179 MainThread heartbeat_tracker INFO     HB stats (seconds): num:43 LIFE_MIN:0.08 min:0.02 mean:0.22 max:1.12 LIFE_MA
    [11/Oct/2020 04:30:52 +0000] 18179 MainThread process      INFO     [120-hbase-MASTER] Updating process.
    ....

[20201014150245430.png_pic_center]: /images/20221123/194522ddb7d04c9e9f30147309f6c63b.png
[https_archives.streamsets.com_datacollector]: https://archives.streamsets.com/datacollector
[https_streamsets.com_documentation_datacollector_3.16.x_help_datacollector_UserGuide_Installation_CMInstall-Overview.html_concept_nb5_c3m_25]: https://streamsets.com/documentation/datacollector/3.16.x/help/datacollector/UserGuide/Installation/CMInstall-Overview.html#concept_nb5_c3m_25