部署CDH以及相关服务启动时的问题与解决方法

灰太狼 2022-07-12 15:16 1171阅读 0赞

部署CDH以及相关服务启动时的问题与解决方法  
（一）系统配置问题  
  （1）磁盘空间不够  
    The I/O cache encountered an error while updating data in medium "ahci-0-0" (rc=VERR\_BAD\_EXE\_FORMAT). Make sure there is enough free space on the disk and that the disk is working properly. Operation can be resumed afterwards.  
   解决：  
 三台机器的配置  
  192.168.1.(164~166)：内存4G，磁盘40G  
  
  
（二）cloudera-scm-server服务  
  （1）cloudera-scm-server启动后马上down掉  
       \# /etc/init.d/cloudera-scm-server status  
       cloudera-scm-server 已死，但 pid 文件仍存    
   解决：  
     1. cloudera-scm-server依赖Mysqld服务，启动必须先启动数据库服务    
      \#/etc/init.d/mysqld start  
      \#chkconfig mysqld on    (实现开机自启动)  
    2. 解析域名vim /etc/hosts   
          IP  名称  
  （2）cloudera-scm-server 启动后挂掉  
     查看日志      
    \# vim /var/log/cloudera-scm-server/cloudera-scm-server.log  
ERROR main:com.cloudera.server.cmf.bootstrap.EntityManagerFactoryBean: Tables have unsupported engine type \[MyISAM\]. InnoDB is required.  
     数据库要求INNODB  
   解决：  
   \# vim /etc/my.cnf  
    \[mysqld\]  
    default-storage-engine=INNODB  
   重启服务  
    \#/etc/init.d/mysld restart  
   重置数据库  
    \#/usr/share/cmf/schema/scm\_prepare\_database.sh mysql -h localhost -uroot -p'123456' --scm-host localhost scm scm scm  
    \#用Cloudera Manager重新建集群  
  
  
（三）安装CDH  
   (1)  yum源报错（安装服务前提前搭建好http服务）  
https://archive.cloudera.com/cm5/redhat/6/x86\_64/cm/5.10.0/repodata/repomd.xml: \[Errno 14\] PYCURL ERROR 6 - "Couldn't resolve host 'archive.cloudera.com'"  
    原因：cloudera-agent的URL不正确  
    解决：搭建http服务器，创建的yum源  
           ![Center][]  
   1. 搭建http服务  
      \#service httpd start  
      \#mkdir /var/www/html/parcel-repo/  
      \#ln -s /opt/cloudera/parcel-repo/  /var/www/html/parcel-repo/  
   2.创建parcel的yum源     
     \#yum install createrepo  -y  
     \#cd /var/www/html/parcel-repo/  
     \#createrepo  .  
3.浏览器测试  
    http://192.168.3.82/parcel-repo/  
  
  
解决cloudera Manager安装锁的问题  
     删掉/tmp下面的  
     \#rm -rf  scm\_prepare\_node.\*  
     \# rm -rf  .scm\_prepare\_node.lock  
   
无法连接HOST\_Monitor的问题（安装完CDH）

（1）确定启动cloudera Managemnt服务

![Center 1][]

（2）agent无法连接HOST\_Monitor的问题  
        1. 查看日志  
         \# cat /var/log/cloudera-scm-agent/cloudera-scm-agent.log  
      INFO CM server guid: d31b5fae-56f4-466e-9f89-d639daa5038b  
        2. 删除cm\_guid  
          \#rm -rf  /var/lib/cloudera-scm-agent/cm\_guid   
        3. 重启服务  
           \#/etc/init.d/cloudera-scm-agent restart  
  guid是agent首次启动时会从server处获得，之后一直保存在agent端。Host-Monitor服务是运行在server端。guid是agent连接server端Host-Monitor服务的一个标识  
  
  
  （3）测某台主机连接不上Host-Monitor  
   Error sending messages to firehose: mgmt-SERVICEMONITOR-ddc9b774d2da234ec1da3086c2a9355f  
  解决：  
     查看是否集群中所有IP与域名对应：vim /etc/hosts  
     重启agent： /etc/init.d/cloudera-scm-agent restart  
  
  
 (4)查询Host-Monitor内部出错  
       Could not fetch descriptor after 5 tries, exiting  
     解决：重启cloudera Manager服务     
  
  
 (5)集群安装报错  
      ERROR  Failure due to stall on seeded torrent.  
    解决：重启agent服务  
        
（五）oozie服务启动报错     
    E0103: Could not load service classes, Cannot load JDBC driver class 'com.mysql.jdbc.Driver'  
    解决：  
     软连接JDBC的驱动器  
    \#ln -s /soft/mysql-connector-java-5.1.40-java.jar /opt/cloudera/parcels/CDH-5.10.0-1.cdh5.10.0.p0.41/lib/oozie/lib/  
  
  
（六）HDFS启动时DataNode异常   
   Initialization failed for Block pool <registering> (Datanode Uuid 526147d4-7dbe-4f8a-acc8-46da63ed8828) service to test1/192.168.1.164:8022. Exiting.   
   解决：  
    \#vim  /dfs/dn/current/VERSION   
      clusterID=cluster6    //保证在同一集群中  
  
  
（七）Hive启动后Hive Meatstore server进程异常退出  
   1）数据库驱动的问题   
       错误1：Error creating transactional connection factory  
          cause by：Attempt to invoke the "BONECP" plugin to create a ConnectionPool gave an error : The specified datastore driver ("com.mysql.jdbc.Driver") was not found in the CLASSPATH. Please check your CLASSPATH specification, and the name of the driver.  
     解决：  
\#ln  -s  /soft/mysql-connector-java-5.1.40-bin.jar  /opt/cloudera/parcels/CDH/lib/hive/auxlib  
  
  
   2）数据库字符问题  
      错误2：Specified key was too long; max key length is 767 bytes  
      原因：Mysql的utf8字符集最大长度限制，注意字符集设置为latin1  
      解决：  
        在CM的web UI界面设置Hive的数据库：

\[Hive\]-->\[配置\]-->\[Hive Metastore数据库\]-->\[Hive Metastore数据库名称\]

![Center 2][]

创建Metastore数据库  
        Mysql>mysql -uroot -p123456  
        mysql>create database hive character set latin1；  
  
  
   3）数据库权限问题  
      错误3：    
       Database access problem. Killing off this connection and all remaining connections in the connection pool. SQL State = HY000  
       解决：  
          grant  all on hive.\* to 'scm'@'192.168.1.164' identified by 'scm';  
          grant  all on hive.\* to 'scm'@'192.168.1.165' identified by 'scm';  
          grant  all on hive.\* to 'scm'@'192.168.1.166' identified by 'scm';  
  
  
   4）错误4：  
     MetaException(message:Version information not found in metastore. )  
     解决：     
       在CM的web UI界面配置Hive服务：  
                ![Center 3][]  
      在hive配置文件中添加：  
      \#cd /opt/cloudera/parcels/CDH/lib/hive/conf  
       \# vim hive-site.xml  
         <property>  
            <name>hive.metastore.schema.verification</name>  
            <value>false</value>  
         </property>

[Center]: /images/20220709/44c42e3c1d5b4edbbaa8707a3ca655a9.png
[Center 1]: /images/20220709/91df7eb9216f46ca90b1218b63fdb521.png
[Center 2]: /images/20220709/e3ede003bbb543bc9a35b44c16854afd.png
[Center 3]: /images/20220709/374b2afe2e064539aafbc39fab6a2b9d.png