Cloudera Manager安装streamsets

ゞ 浴缸里的玫瑰 2022-12-14 08:52 362阅读 0赞

文章目录

  • 1, 下载streamsets的parcel文件
  • 2, 使用cloudera manager安装parcel
    • a, 分发parcel报错
      • 报错后，下载好的parcel被删除
    • b, 错误解决

在这里插入图片描述

1, 下载streamsets的parcel文件

parcel下载地址： https://archives.streamsets.com/datacollector
parcel使用cloudera manager安装文档：https://streamsets.com/documentation/datacollector/3.16.x/help/datacollector/UserGuide/Installation/CMInstall-Overview.html#concept_nb5_c3m_25

  • Install the StreamSets custom service descriptor (CSD). (需要重启cm_server, cm监控服务)
  • (Optional.) Manually install the parcel and checksum files. Typically only needed when the Cloudera Manager Server does not have internet access.
  • Download, distribute, and activate the StreamSets parcel.

2, 使用cloudera manager安装parcel

a, 分发parcel报错

  • 问题1: host_monitor退出,
  • 问题2: parcel下载后解压失败被删除

  • 问题3: 分发parcel卡住

    =================第一次：agent, 下载解压报错
    [root@test-c6 ~]# find /opt/ -name “STREAM*”
    /opt/cloudera/parcel-repo/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel
    /opt/cloudera/parcel-repo/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel.torrent
    /opt/cloudera/parcel-repo/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel.sha
    /opt/cloudera/csd/STREAMSETS-3.16.2.jar
    /opt/cloudera/parcel-cache/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel.torrent
    /opt/cloudera/parcels/.flood/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel
    /opt/cloudera/parcels/.flood/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel
    /opt/cloudera/parcels/.flood/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel.torrent
    /opt/cloudera/parcels/STREAMSETS_DATACOLLECTOR
    /opt/cloudera/parcels/STREAMSETS_DATACOLLECTOR-3.16.2

    [root@test-c6 parcel-repo]# tailf /opt/cm-5.12.2/log/cloudera-scm-agent/cloudera-scm-agent.log
    [11/Oct/2020 01:36:34 +0000] 28799 Thread-13 downloader INFO Current state: STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel [totalDownloaded=5611142ize=5712330086 upload=0 state=downloading seed=[‘http://test-c6:7180/cmf/parcel/download/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel‘] location=/opt/crcels/.flood/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel progress=982286]
    [11/Oct/2020 01:36:36 +0000] 28799 Thread-13 downloader INFO Current state: STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel [totalDownloaded=5712330ize=5712330086 upload=0 state=seeding seed=[‘http://test-c6:7180/cmf/parcel/download/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel‘] location=/opt/clouds/.flood/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel progress=1000000]
    [11/Oct/2020 01:36:36 +0000] 28799 Thread-13 downloader INFO Completed download of http://test-c6:7180/cmf/parcel/download/STREAMSETS_DATACOLLE2-el6.parcel code=200 state=downloaded
    [11/Oct/2020 01:36:36 +0000] 28799 Thread-13 parcel_cache WARNING No checksum in header, skipping verification
    [11/Oct/2020 01:36:36 +0000] 28799 Thread-13 parcel_cache INFO Unpacking /opt/cloudera/parcels/.flood/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel/_DATACOLLECTOR-3.16.2-el6.parcel into /opt/cloudera/parcels
    [11/Oct/2020 01:37:08 +0000] 28799 MainThread agent INFO PID ‘30453’ associated with process ‘124-cloudera-mgmt-SERVICEMONITOR’ with paylosname:124-cloudera-mgmt-SERVICEMONITOR groupname:124-cloudera-mgmt-SERVICEMONITOR from_state:RUNNING expected:0 pid:30453’ exited unexpectedly [11/Oct/2020 01:37:34 +0000] 28799 Monitor-GenericMonitor throttling_logger ERROR Error fetching metrics at ‘http://test-c6.com:8091/jmx‘ Traceback (most recent call last): File “/opt/cm-5.12.2/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.12.2-py2.6.egg/cmf/monitor/generic/metric_collectors.py”, line 200ect_and_parse_and_return self._adapter.safety_valve)) File “/opt/cm-5.12.2/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.12.2-py2.6.egg/cmf/url_util.py”, line 207, in urlopen_with_retry_ocation_errors return function() File “/opt/cm-5.12.2/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.12.2-py2.6.egg/cmf/monitor/generic/metric_collectors.py”, line 217_url password=self._password_value) File “/opt/cm-5.12.2/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.12.2-py2.6.egg/cmf/url_util.py”, line 70, in urlopen_with_timeout return opener.open(url, data, timeout) File “/usr/lib64/python2.6/urllib2.py”, line 391, in open response = self._open(req, data) File “/usr/lib64/python2.6/urllib2.py”, line 409, in _open ‘_open’, req) File “/usr/lib64/python2.6/urllib2.py”, line 369, in _call_chain result = func(*args) File “/usr/lib64/python2.6/urllib2.py”, line 1190, in http_open return self.do_open(httplib.HTTPConnection, req) File “/usr/lib64/python2.6/urllib2.py”, line 1165, in do_open raise URLError(err) URLError: [11/Oct/2020 01:37:35 +0000] 28799 MonitorDaemon-Reporter proc_metrics_utils ERROR Failed to read file descriptor max for process 30381: [Errno 2]ile or directory: ‘/proc/30381/limits’ [11/Oct/2020 01:37:35 +0000] 28799 MonitorDaemon-Reporter proc_metrics_utils ERROR Failed to get file descriptor count for process 30381: [Errno 2file or directory: ‘/proc/30381/fd/‘ [11/Oct/2020 01:37:35 +0000] 28799 MonitorDaemon-Reporter proc_metrics_utils ERROR Failed to get process metrics 30381: no process found with pid [11/Oct/2020 01:37:36 +0000] 28799 MonitorDaemon-Reporter throttling_logger ERROR Error sending messages to firehose: mgmt-SERVICEMONITOR-af401409538410569c49e2 Traceback (most recent call last): File “/opt/cm-5.12.2/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.12.2-py2.6.egg/cmf/monitor/firehose.py”, line 120, in _send self._requestor.request(‘sendAgentMessages’, dict(messages=messages)) File “/opt/cm-5.12.2/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py”, line 141, in request return self.issue_request(call_request, message_name, request_datum) File “/opt/cm-5.12.2/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py”, line 254, in issue_request call_response = self.transceiver.transceive(call_request) File “/opt/cm-5.12.2/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py”, line 482, in transceive self.write_framed_message(request) File “/opt/cm-5.12.2/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py”, line 501, in write_framed_message self.conn.request(req_method, self.req_resource, req_body, req_headers) File “/usr/lib64/python2.6/httplib.py”, line 979, in request self._send_request(method, url, body, headers) File “/usr/lib64/python2.6/httplib.py”, line 1010, in _send_request self.endheaders() File “/usr/lib64/python2.6/httplib.py”, line 967, in endheaders self._send_output() File “/usr/lib64/python2.6/httplib.py”, line 831, in _send_output self.send(msg) File “/usr/lib64/python2.6/httplib.py”, line 790, in send self.connect() File “/usr/lib64/python2.6/httplib.py”, line 771, in connect self.timeout) File “/usr/lib64/python2.6/socket.py”, line 567, in create_connection raise error, msg error: [Errno 111] Connection refused [11/Oct/2020 01:37:36 +0000] 28799 MonitorDaemon-Reporter throttling_logger ERROR Error sending messages to firehose: mgmt-HOSTMONITOR-af401409b8a410569c49e2 Traceback (most recent call last): File “/opt/cm-5.12.2/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.12.2-py2.6.egg/cmf/monitor/firehose.py”, line 120, in _send self._requestor.request(‘sendAgentMessages’, dict(messages=messages)) File “/opt/cm-5.12.2/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py”, line 141, in request return self.issue_request(call_request, message_name, request_datum) File “/opt/cm-5.12.2/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py”, line 254, in issue_request call_response = self.transceiver.transceive(call_request) File “/opt/cm-5.12.2/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py”, line 482, in transceive self.write_framed_message(request) File “/opt/cm-5.12.2/lib64/cmf/agent/build/env/lib/python2.6/site-packages/avro-1.6.3-py2.6.egg/avro/ipc.py”, line 501, in write_framed_message self.conn.request(req_method, self.req_resource, req_body, req_headers) File “/usr/lib64/python2.6/httplib.py”, line 979, in request self._send_request(method, url, body, headers) File “/usr/lib64/python2.6/httplib.py”, line 1010, in _send_request self.endheaders() File “/usr/lib64/python2.6/httplib.py”, line 967, in endheaders self._send_output() File “/usr/lib64/python2.6/httplib.py”, line 831, in _send_output self.send(msg) File “/usr/lib64/python2.6/httplib.py”, line 790, in send self.connect() File “/usr/lib64/python2.6/httplib.py”, line 771, in connect self.timeout) File “/usr/lib64/python2.6/socket.py”, line 567, in create_connection raise error, msg error: [Errno 111] Connection refused [11/Oct/2020 01:37:36 +0000] 28799 MainThread agent INFO PID ‘30381’ associated with process ‘121-cloudera-mgmt-HOSTMONITOR’ with payload me:121-cloudera-mgmt-HOSTMONITOR groupname:121-cloudera-mgmt-HOSTMONITOR from_state:RUNNING expected:0 pid:30381’ exited unexpectedly
    [11/Oct/2020 01:38:03 +0000] 28799 MainThread agent INFO PID ‘30243’ associated with process ‘121-cloudera-mgmt-HOSTMONITOR’ with payload me:121-cloudera-mgmt-HOSTMONITOR groupname:121-cloudera-mgmt-HOSTMONITOR from_state:RUNNING expected:0 pid:30243’ exited unexpectedly [11/Oct/2020 01:38:07 +0000] 28799 MainThread agent INFO PID ‘30316’ associated with process ‘124-cloudera-mgmt-SERVICEMONITOR’ with paylosname:124-cloudera-mgmt-SERVICEMONITOR groupname:124-cloudera-mgmt-SERVICEMONITOR from_state:RUNNING expected:0 pid:30316’ exited unexpectedly
    [11/Oct/2020 01:38:28 +0000] 28799 MainThread agent INFO PID ‘30416’ associated with process ‘121-cloudera-mgmt-HOSTMONITOR’ with payload me:121-cloudera-mgmt-HOSTMONITOR groupname:121-cloudera-mgmt-HOSTMONITOR from_state:RUNNING expected:0 pid:30416’ exited unexpectedly

报错后，下载好的parcel被删除

  1. #=================之后, /opt/cloudera/parcels/.flood/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel文件被删除, 并开始报下面的错
  2. [root@test-c6 parcel-repo]# find /opt/ -name "STREAM*"
  3. /opt/cloudera/parcel-repo/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel
  4. /opt/cloudera/parcel-repo/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel.torrent
  5. /opt/cloudera/parcel-repo/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel.sha
  6. /opt/cloudera/csd/STREAMSETS-3.16.2.jar
  7. /opt/cloudera/parcel-cache/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel.torrent
  8. /opt/cloudera/parcels/.flood/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel
  9. /opt/cloudera/parcels/.flood/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel.torrent
  11. [root@test-c6 parcel-repo]# tailf /opt/cm-5.12.2/log/cloudera-scm-agent/cloudera-scm-agent.log
  12. [11/Oct/2020 01:59:47 +0000] 1594 Thread-13 downloader   INFO     Fetching torrent: http://test-c6:7180/cmf/parcel/download/STREAMSETS_DATACOLLECTOR-.parcel.torrent
  13. [11/Oct/2020 01:59:47 +0000] 1594 Thread-13 downloader   INFO     Starting download of: http://test-c6:7180/cmf/parcel/download/STREAMSETS_DATACOLLEC-el6.parcel
  14. [11/Oct/2020 01:59:47 +0000] 1594 Thread-13 downloader   INFO     Failed adding torrent: file:///opt/cloudera/parcel-cache/STREAMSETS_DATACOLLECTOR-3parcel.torrent Already present torrent: STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel
  15. [11/Oct/2020 01:59:47 +0000] 1594 Thread-13 downloader   INFO     Current state: STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel [totalDownloaded=57123300ze=5712330086 upload=0 state=seeding seed=['http://test-c6:7180/cmf/parcel/download/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel'] location=/opt/cloude/.flood/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel progress=1000000]
  16. [11/Oct/2020 01:59:47 +0000] 1594 Thread-13 downloader   INFO     Completed download of http://test-c6:7180/cmf/parcel/download/STREAMSETS_DATACOLLEC-el6.parcel code=200 state=downloaded
  17. [11/Oct/2020 01:59:47 +0000] 1594 Thread-13 parcel_cache WARNING  No checksum in header, skipping verification
  18. [11/Oct/2020 01:59:47 +0000] 1594 Thread-13 parcel_cache INFO     Unpacking /opt/cloudera/parcels/.flood/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel/SDATACOLLECTOR-3.16.2-el6.parcel into /opt/cloudera/parcels
  19. [11/Oct/2020 01:59:47 +0000] 1594 Thread-13 downloader   ERROR    Failed op: Src file /opt/cloudera/parcels/.flood/STREAMSETS_DATACOLLECTOR-3.16.2-elTREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel does not exist
  20. Traceback (most recent call last):
  21.   File "/opt/cm-5.12.2/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.12.2-py2.6.egg/cmf/downloader.py", line 501, in callable
  22.     callback(url, curr_op)
  23.   File "/opt/cm-5.12.2/lib64/cmf/agent/build/env/lib/python2.6/site-packages/cmf-5.12.2-py2.6.egg/cmf/parcel_cache.py", line 203, in cb
  24.     raise e
  25. Exception: Src file /opt/cloudera/parcels/.flood/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel does not exist
  26. [11/Oct/2020 01:59:47 +0000] 1594 Thread-13 downloader   INFO     Finished download [ url: http://test-c6:7180/cmf/parcel/download/STREAMSETS_DATACOL6.2-el6.parcel, state: exception, total_bytes: 5712330086, downloaded_bytes: 5712330086, start_time: 2020-10-11 01:59:47, download_end_time: 2020-10-7, end_time: 2020-10-11 01:59:47, code: 601, exception_msg: Src file /opt/cloudera/parcels/.flood/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel/STREAMSELECTOR-3.16.2-el6.parcel does not exist, path: /opt/cloudera/parcels/.flood/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel/STREAMSETS_DATACOLLECTOR-3.16.el ]

b, 错误解决

  • 第一步: host_monitor,service_monitor 配置修改: jvm内存=1g, 文件描述符数=65536
  • 第二步: 系统参数修改 /etc/security/limits.d/的nofile值=65536, 重启sshd

  • 第三步: 重启监控服务, 重启agent （触发再次下载parcel)

    =================第二次：agent重启(修改host_monitor,service_monitor 的jvm参数1g,

    1.                 Cloudera Management Service的最大进程文件描述符数=65536, 
    2.                 和/etc/security/limits.d/的nofile值=65536, 重启sshd, 
    3.                 重启监控服务, 重启agent) 
    4.         下载解压正常

    [11/Oct/2020 04:20:03 +0000] 18179 Thread-13 downloader INFO Current state: STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel [totalDownloaded=56169095712330086 upload=0 state=downloading seed=[‘http://test-c6:7180/cmf/parcel/download/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel‘] location=/opt/cloudlood/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel progress=983295]
    [11/Oct/2020 04:20:05 +0000] 18179 Thread-13 downloader INFO Current state: STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel [totalDownloaded=57123305712330086 upload=0 state=seeding seed=[‘http://test-c6:7180/cmf/parcel/download/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel‘] location=/opt/cloudera//STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel progress=1000000]
    [11/Oct/2020 04:20:05 +0000] 18179 Thread-13 downloader INFO Completed download of http://test-c6:7180/cmf/parcel/download/STREAMSETS_DATACOLLE6.parcel code=200 state=downloaded
    [11/Oct/2020 04:20:05 +0000] 18179 Thread-13 parcel_cache WARNING No checksum in header, skipping verification
    [11/Oct/2020 04:20:05 +0000] 18179 Thread-13 parcel_cache INFO Unpacking /opt/cloudera/parcels/.flood/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel/ACOLLECTOR-3.16.2-el6.parcel into /opt/cloudera/parcels
    [11/Oct/2020 04:21:43 +0000] 18179 MainThread agent WARNING Long HB processing time: 6.20692515373

    [11/Oct/2020 04:22:20 +0000] 18179 Thread-13 parcel_cache INFO Unpack of parcel /opt/cloudera/parcels/.flood/STREAMSETS_DATACOLLECTOR-3.16.2-el6.ETS_DATACOLLECTOR-3.16.2-el6.parcel successful
    [11/Oct/2020 04:22:20 +0000] 18179 Thread-13 downloader INFO Finished download [ url: http://test-c6:7180/cmf/parcel/download/STREAMSETS_DATACO-el6.parcel, state: complete, total_bytes: 5712330086, downloaded_bytes: 5712330086, start_time: 2020-10-11 04:15:14, download_end_time: 2020-10-11 0ime: 2020-10-11 04:22:20, code: 200, exception_msg: None, path: /opt/cloudera/parcels/.flood/STREAMSETS_DATACOLLECTOR-3.16.2-el6.parcel/STREAMSETS_DA16.2-el6.parcel ]
    [11/Oct/2020 04:22:22 +0000] 18179 MainThread parcel INFO Loading parcel manifest for: CDH-5.12.0-1.cdh5.12.0.p0.29
    [11/Oct/2020 04:22:22 +0000] 18179 MainThread parcel INFO Loading parcel manifest for: STREAMSETS_DATACOLLECTOR-3.16.2
    [11/Oct/2020 04:22:22 +0000] 18179 MainThread parcel INFO Ensuring users/groups exist for new parcel STREAMSETS_DATACOLLECTOR-3.16.2.
    [11/Oct/2020 04:22:22 +0000] 18179 MainThread parcel INFO Executing command [‘/usr/sbin/groupadd’, ‘-r’, ‘sdc’]
    [11/Oct/2020 04:22:22 +0000] 18179 MainThread parcel INFO Executing command [‘/usr/sbin/groupadd’, ‘-r’, ‘sdc’]
    [11/Oct/2020 04:22:22 +0000] 18179 MainThread parcel INFO Executing command [‘/usr/sbin/useradd’, ‘-r’, ‘-m’, ‘-g’, ‘sdc’, ‘-K’, ‘UMASK=022/var/lib/sdc’, ‘—comment’, ‘StreamSets Data Collector’, ‘—shell’, ‘/sbin/nologin’, ‘sdc’]
    [11/Oct/2020 04:22:22 +0000] 18179 MainThread parcel INFO Ensuring correct file permissions for new parcel STREAMSETS_DATACOLLECTOR-3.16.2.
    [11/Oct/2020 04:25:22 +0000] 18179 MainThread heartbeat_tracker INFO HB stats (seconds): num:43 LIFE_MIN:0.08 min:0.02 mean:0.22 max:1.12 LIFE_MA
    [11/Oct/2020 04:30:52 +0000] 18179 MainThread process INFO [120-hbase-MASTER] Updating process.
    ….

发表评论

表情：
评论列表 （有 0 条评论，362人围观）

还没有评论，来说两句吧...

相关阅读

    相关 安装Cloudera Manager

    cloudera manager有四大功能： （1）管理：对集群进行管理，如添加、删除节点等操作。 （2）监控：监控集群的健康情况，对设置的各种指标和系统运行

    你的名字你的名字/ 0 赞/ 335 阅读