Spark、Hadoop大数据平台搭建

深碍√TFBOYSˉ_ 2022-04-10 03:17 220阅读 0赞

## 下载安装包 ##

**Spark**

分布式计算

[spark-2.3.2-bin-hadoop2.7][]，安装包大小：220M

支持Hadoop 2.7以后的版本

**Scala**

Scala环境，Spark的开发语言

[scala-2.12.8.tgz][]，安装包大小：20M

**Hadoop**

分布式存储（计算）

[hadoop-2.7.7.tar.gz][]，安装包大小：209M

**Java**

Java环境

[Java SE Development Kit 8u192][]，安装包大小：187M

**ZooKeeper**

注册中心

[zookeeper-3.4.13.tar.gz][]，安装包大小：36M

## 配置 ##

首先需要配置环境变量

### 环境变量 ###

打开并编辑`~/.bashrc`文件，在末尾添加以下内容：

# java
    JAVA_HOME=/opt/jdk1.8.0_192
    export PATH=$PATH:$JAVA_HOME/bin
    
    # hadoop
    HADOOP_HOME=/opt/hadoop-2.7.7
    export PATH=$PATH:$HADOOP_HOME/bin
    
    # zookeeper
    ZOOKEEPER_HOME=/opt/zookeeper-3.4.13
    export PATH=$PATH:$ZOOKEEPER_HOME/bin
    
    # scala
    SCALA_HOME=/opt/scala-2.12.8
    export PATH=$PATH:$SCALA_HOME/bin
    
    # spark
    SPARK_HOME=/opt/spark-2.3.2-bin-hadoop2.7
    export PATH=$PATH:$SPARK_HOME/bin

### Host ###

这里需要设置Host，一个是Hadoop的Name-Node、Data-Node，一个spark的Master、Worker。

# hadoop
    192.168.1.151 name-node
    192.168.1.155 data-node-sa
    192.168.1.156 data-node-sb
    192.168.1.157 data-node-sc

### spark ###

**conf/slaves**

worker-sa
    worker-sb
    worker-sc

待补充 。。。

### hadoop ###

**etc/hadoop/slaves**

data-node-sa
    data-node-sb
    data-node-sc

## Python3安装 ##

**安装pyspark**

cd /opt/spark-2.3.2-bin-hadoop2.7/python && python36 setup.py install

**安装numpy**

pip3 install numpy -i https://pypi.mirrors.ustc.edu.cn/simple/ --trusted-host pypi.mirrors.ustc.edu.cn

## 启动&停止 ##

hadoop

sbin/start-all.sh
    sbin/stop-all.sh

spark

sbin/start-all.sh
    sbin/stop-all.sh

## 执行任务 ##

**提交任务**

spark-submit --master spark://name-node:7077 --executor-memory 512m --total-executor-cores 2 client.py

## 几个管理页 ##

**Spark**

[http://name-node:8080/][http_name-node_8080]  
[http://name-node:4040/（执行任务时有效）][http_name-node_4040]

**Hadoop**

[http://name-node:8088/][http_name-node_8088]  
[http://name-node:50070/（可查看集群内的文件列表）][http_name-node_50070]

## 测试数据 ##

[http://files.grouplens.org/datasets/movielens/][http_files.grouplens.org_datasets_movielens]  
[http://files.grouplens.org/datasets/movielens/ml-10m.zip(ratings.dat)][http_files.grouplens.org_datasets_movielens_ml-10m.zip_ratings.dat]

[spark-2.3.2-bin-hadoop2.7]: http://mirrors.shu.edu.cn/apache/spark/spark-2.3.2/spark-2.3.2-bin-hadoop2.7.tgz
[scala-2.12.8.tgz]: https://downloads.lightbend.com/scala/2.12.8/scala-2.12.8.tgz
[hadoop-2.7.7.tar.gz]: https://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/hadoop-2.7.7/hadoop-2.7.7.tar.gz
[Java SE Development Kit 8u192]: https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
[zookeeper-3.4.13.tar.gz]: https://mirrors.tuna.tsinghua.edu.cn/apache/zookeeper/zookeeper-3.4.13/zookeeper-3.4.13.tar.gz
[http_name-node_8080]: http://name-node:8080/
[http_name-node_4040]: http://name-node:4040/%EF%BC%88%E6%89%A7%E8%A1%8C%E4%BB%BB%E5%8A%A1%E6%97%B6%E6%9C%89%E6%95%88%EF%BC%89
[http_name-node_8088]: http://name-node:8088/
[http_name-node_50070]: http://name-node:50070/%EF%BC%88%E5%8F%AF%E6%9F%A5%E7%9C%8B%E9%9B%86%E7%BE%A4%E5%86%85%E7%9A%84%E6%96%87%E4%BB%B6%E5%88%97%E8%A1%A8%EF%BC%89
[http_files.grouplens.org_datasets_movielens]: http://files.grouplens.org/datasets/movielens/
[http_files.grouplens.org_datasets_movielens_ml-10m.zip_ratings.dat]: http://files.grouplens.org/datasets/movielens/ml-10m.zip%28ratings.dat%29