Hadoop 集群常用脚本

水深无声 2022-08-30 04:06 149阅读 0赞

## 一、集群分发脚本实现 ##

**1、实现功能**

循环复制文件到所有集群节点的相同目录下。

**2、具体操作**

(1) 在/home/yasin/bin 目录下创建 xsync 文件

[yasin@hadoop102 opt]$ cd /home/yasin
    [yasin@hadoop102 ~]$ mkdir bin
    [yasin@hadoop102 ~]$ cd bin
    [yasin@hadoop102 bin]$ vim xsync

实现脚本：

#!/bin/bash
    #1. 判断参数个数
    if [ $# -lt 1 ]
    then
            echo Not Enough Arguement!
            exit;
    fi
    
    #2. 遍历集群所有机器（根据自己的集群名修改）
    for host in hadoop102 hadoop103 hadoop104
    do
            echo ==================== $host ====================
            #3. 遍历所有目录，挨个发送
            for file in $@
            do
                    #4. 判断文件是否存在
                    if [ -e $file ]
                            then
                                    #5. 获取父目录
                                    pdir=$(cd -P $(dirname $file); pwd)
                                    #6. 获取当前文件的名称
                                    fname=$(basename $file)
                                    ssh $host "mkdir -p $pdir"
                                    rsync -av $pdir/$fname $host:$pdir
                            else
                                    echo $file does not exists!
                    fi
            done
    done

（2）修改脚本 xsync 具有执行权限

[yasin@hadoop102 bin]$ chmod +x xsync

（3）测试脚本

[yasin@hadoop102 ~]$ xsync /home/yasin/bin

（4）将脚本复制到/bin 中，以便全局调用

[yasin@hadoop102 bin]$ sudo cp xsync /bin/

**3、实例操作**

[yasin@haddoop102 hadoop]$ xsync yarn-site.xml 
    ==================== hadoop102 ====================
    sending incremental file list
    yarn-site.xml
    
    sent 1,042 bytes  received 47 bytes  726.00 bytes/sec
    total size is 1,626  speedup is 1.49
    ==================== hadoop103 ====================
    sending incremental file list
    
    sent 65 bytes  received 12 bytes  51.33 bytes/sec
    total size is 1,626  speedup is 21.12
    ==================== hadoop104 ====================
    sending incremental file list
    yarn-site.xml

## 二、 Hadoop 集群启停脚本（包含 HDFS ，Yarn ，Historyserver ） ##

**1、实现功能**  
Hadoop 集群批量启停脚本（包含 HDFS ，Yarn ，Historyserver ）

[yasin@hadoop102 ~]$ cd /home/yasin/bin
    [yasin@hadoop102 bin]$ vim myhadoop.sh

**2、具体操作**  
实现脚本（根据自己的节点名称和路径相应修改）：

#!/bin/bash
    if [ $# -lt 1 ]
    then
            echo "No Args Input..."
            exit ;
    fi
    case $1 in
            "start")
            echo " =================== 启动 hadoop 集群 ==================="
            echo " --------------- 启动 hdfs ---------------"
            ssh hadoop102 "/opt/module/hadoop-3.3.1/sbin/start-dfs.sh"
            echo " --------------- 启动 yarn ---------------"
            ssh hadoop103 "/opt/module/hadoop-3.3.1/sbin/start-yarn.sh"
            echo " --------------- 启动 historyserver ---------------"
            ssh hadoop102 "/opt/module/hadoop-3.3.1/bin/mapred --daemon start historyserver"
    ;;
            "stop")
            echo " =================== 关闭 hadoop 集群 ==================="
            echo " --------------- 关闭 historyserver ---------------"
            ssh hadoop102 "/opt/module/hadoop-3.3.1/bin/mapred --daemon stop historyserver"
            echo " --------------- 关闭 yarn ---------------"
            ssh hadoop103 "/opt/module/hadoop-3.3.1/sbin/stop-yarn.sh"
            echo " --------------- 关闭 hdfs ---------------"
            ssh hadoop102 "/opt/module/hadoop-3.3.1/sbin/stop-dfs.sh"
    ;;
    *)
            echo "Input Args Error..."
    ;;
    esac

保存后退出，然后赋予脚本执行权限

[yasin@hadoop102 bin]$ chmod +x myhadoop.sh

**3、实例操作**  
（1）全部停止

[yasin@haddoop102 bin]$ myhadoop.sh stop
     =================== 关闭 hadoop 集群 ===================
     --------------- 关闭 historyserver ---------------
     --------------- 关闭 yarn ---------------
    Stopping nodemanagers
    Stopping resourcemanager
     --------------- 关闭 hdfs ---------------
    Stopping namenodes on [hadoop102]
    Stopping datanodes
    Stopping secondary namenodes [hadoop104]

（2）全部开启

[yasin@haddoop102 bin]$ myhadoop.sh start
     =================== 启动 hadoop 集群 ===================
     --------------- 启动 hdfs ---------------
    Starting namenodes on [hadoop102]
    Starting datanodes
    Starting secondary namenodes [hadoop104]
     --------------- 启动 yarn ---------------
    Starting resourcemanager
    Starting nodemanagers
     --------------- 启动 historyserver ---------------

## 三、查看多台服务器 Java 进程脚本 ##

**1、实现功能**  
实现一次查看三台服务器 Java 进程

**2、具体操作**

[yasin@hadoop102 ~]$ cd /home/yasin/bin
    [yasin@hadoop102 bin]$ vim jpsall

实现脚本：

#!/bin/bash
    for host in hadoop102 hadoop103 hadoop104
    do
            echo =============== $host ===============
            ssh $host jps
    done

保存后退出，然后赋予脚本执行权限  
\[yasin@hadoop102 bin\]$ chmod +x jpsall

**3、实例操作**

[yasin@haddoop102 bin]$ jpsall
    =============== hadoop102 ===============
    27173 JobHistoryServer
    26679 DataNode
    27001 NodeManager
    27355 Jps
    26559 NameNode
    =============== hadoop103 ===============
    20677 DataNode
    21416 Jps
    20857 ResourceManager
    20975 NodeManager
    =============== hadoop104 ===============
    19761 NodeManager
    19667 SecondaryNameNode
    19975 Jps
    19566 DataNode

**注：本博客中大数据相关内容均来自[尚硅谷大数据课程@大海哥][Link 1]。其他大数据博客中未加本注释，默认已加本声明，主要为了学习和参考。**

[Link 1]: https://www.bilibili.com/video/BV1Qp4y1n7EN