hive on spark编译

た入场券 2022-10-29 09:22 192阅读 0赞

# 一.前置条件 #

1.  官网下载Spark源码  
    ![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQ2NTQ4ODU1_size_16_color_FFFFFF_t_70][]  
    Spark2.2.0版本文档地址：[https://spark.apache.org/docs/2.4.5/][https_spark.apache.org_docs_2.4.5]  
    必备条件：Maven requires Maven 3.5.4 and Java 8  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQ2NTQ4ODU1_size_16_color_FFFFFF_t_70 1]

## 安装maven ##

链接：https://pan.baidu.com/s/13sVa3Z6SktogXDlnNe89Yw 
    提取码：iuja

1.解压maven

tar -zxvf apache-maven-3.6.1-bin.tar.gz -C /export/servers/

2.添加环境变量,在profile文件结尾添加以下内容

vim /etc/profile

#MAVEN_HOME
    export MAVEN_HOME=/export/servers/apache-maven-3.6.1
    export PATH=$PATH:$MAVEN_HOME/bin

source /etc/profile

3.测试安装结果

[root@hadoop102 conf]# mvn -v
    Apache Maven 3.6.1 (d66c9c0b3152b2e69ee9bac180bb8fcc8e6af555; 2019-04-05T03:00:29+08:00)
    Maven home: /export/servers/apache-maven-3.6.1
    Java version: 1.8.0_162, vendor: Oracle Corporation, runtime: /export/servers/jdk/jre
    Default locale: en_US, platform encoding: UTF-8
    OS name: "linux", version: "3.10.0-1062.el7.x86_64", arch: "amd64", family: "unix"

4.配置阿里云镜像,注意添加miroor镜像一定要在mirrors标签内，否则无效

vi /export/servers/apache-maven-3.6.1/conf/settings.xml

<mirror>
            <id>nexus-aliyun</id>
            <mirrorOf>central</mirrorOf>
            <name>Nexus aliyun</name>
            <url>http://maven.aliyun.com/nexus/content/groups/public</url>
    </mirror>

## 编译步骤 ##

建议看看hive和hadoop的对应版本是否正确(我之前使用hadoop2.7.7 hive3.1.2 spark2.4.5  
整不起 原因hive和hadoop版本不对应)  
[https://blog.csdn.net/qq\_46548855/article/details/113792347][https_blog.csdn.net_qq_46548855_article_details_113792347]  
环境hadoop-3.1.3 hive-3.1.2  
对于我们使用2.4.5的spark需要进行编译  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQ2NTQ4ODU1_size_16_color_FFFFFF_t_70 2]  
执行编译命令

./dev/make-distribution.sh --name without-hive --tgz -Pyarn -Phadoop-3.1 -Dhadoop.version=3.1.3 -Pparquet-provided -Porc-provided -Phadoop-provided

如果执行这段代码没反应

vi /spark-2.4.5/dev/make-distribution.sh

删除里面这几行代码，再次执行

128 VERSION=$("$MVN" help:evaluate -Dexpression=project.version $@ 2>/dev/null\
        129     | grep -v "INFO"\
        130     | grep -v "WARNING"\
        131     | tail -n 1)
        132 SCALA_VERSION=$("$MVN" help:evaluate -Dexpression=scala.binary.version $@ 2>/dev/null\
        133     | grep -v "INFO"\
        134     | grep -v "WARNING"\
        135     | tail -n 1)
        136 SPARK_HADOOP_VERSION=$("$MVN" help:evaluate -Dexpression=hadoop.version $@ 2>/dev/null\
        137     | grep -v "INFO"\
        138     | grep -v "WARNING"\
        139     | tail -n 1)
        141     | grep -v "INFO"\
        142     | grep -v "WARNING"\
        143     | fgrep --count "<id>hive</id>";\
        144     # Reset exit status to 0, otherwise the script stops here if the last grep finds nothing\
        145     # because we use "set -o pipefail"
        146     echo -n)

下载会让你下载两个文件非常的慢，直接复制网址，在window上下载之后上传到  
/spark-2.4.5/build下  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQ2NTQ4ODU1_size_16_color_FFFFFF_t_70 3]

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQ2NTQ4ODU1_size_16_color_FFFFFF_t_70 4]  
**如果失败了，多试几次，原因网速问题**  
等待编译完成，spark-2.4.5-bin-without-hive.tgz为最终文件

链接：https://pan.baidu.com/s/19K4BZfsQ50ziFyfYeU36nw 
    提取码：dl8t

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQ2NTQ4ODU1_size_16_color_FFFFFF_t_70]: /images/20221024/79be18d6d99e4e14b4815b4398d2e012.png
[https_spark.apache.org_docs_2.4.5]: https://spark.apache.org/docs/2.4.5/
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQ2NTQ4ODU1_size_16_color_FFFFFF_t_70 1]: /images/20221024/6a47959e2d4043b18194e863e0553fb5.png
[https_blog.csdn.net_qq_46548855_article_details_113792347]: https://blog.csdn.net/qq_46548855/article/details/113792347
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQ2NTQ4ODU1_size_16_color_FFFFFF_t_70 2]: /images/20221024/cad131c603604a71b725bd13683ed768.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQ2NTQ4ODU1_size_16_color_FFFFFF_t_70 3]: /images/20221024/84876df98fc24ed4bb6f6f99fe833901.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQ2NTQ4ODU1_size_16_color_FFFFFF_t_70 4]: /images/20221024/025462fcc5324521ab05544d5d8c02a0.png