mac搭建大数据开发环境

叁歲伎倆 2022-09-02 04:09 249阅读 0赞

mac安装Hadoop/hbase/spark：[https://blog.csdn.net/sunxiaoju/article/details/86183405][https_blog.csdn.net_sunxiaoju_article_details_86183405]

mac idea+sbt运行spark程序：[https://blog.csdn.net/sunxiaoju/article/details/100529569][https_blog.csdn.net_sunxiaoju_article_details_100529569]

mac安装hive：[https://blog.csdn.net/sunxiaoju/article/details/102247369][https_blog.csdn.net_sunxiaoju_article_details_102247369]

spark查询hive：[https://blog.csdn.net/weixin\_44131414/article/details/102611191][https_blog.csdn.net_weixin_44131414_article_details_102611191]

踩坑：

1.提示无法连接thrift://master:9083

将hive-site.xml中的master:9083改为127.0.0.1:9083

2.提示Table or view not found: employee; line 1 pos 14

将代码spark.sql(sql2).show()改为spark.sql(sql).show()

spark查询mysql：[https://blog.csdn.net/sunxiaoju/article/details/103793284][https_blog.csdn.net_sunxiaoju_article_details_103793284]

踩坑：

1.sparkSession无法导入，在maven pom.xml文件中加入如下配置

<groupId>org.apache.spark</groupId>

<artifactId>spark-sql\_2.11</artifactId>

</dependency>

2.运行报错Exception in thread "main" java.lang.ClassNotFoundException: com.mysql.cj.jd

将jar包mysql-connector-java-8.0.26.jar导入到项目中

3.运行报错Exception in thread "main" com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure

将代码中的mysql://master:3306/hive改为mysql://127.0.0.1:3306/hive

[https_blog.csdn.net_sunxiaoju_article_details_86183405]: https://blog.csdn.net/sunxiaoju/article/details/86183405
[https_blog.csdn.net_sunxiaoju_article_details_100529569]: https://blog.csdn.net/sunxiaoju/article/details/100529569
[https_blog.csdn.net_sunxiaoju_article_details_102247369]: https://blog.csdn.net/sunxiaoju/article/details/102247369
[https_blog.csdn.net_weixin_44131414_article_details_102611191]: https://blog.csdn.net/weixin_44131414/article/details/102611191
[https_blog.csdn.net_sunxiaoju_article_details_103793284]: https://blog.csdn.net/sunxiaoju/article/details/103793284