minikube运行sparkPi

男娘i 2022-12-24 12:52 70阅读 0赞

spark-on-k8s是spark执行任务的一种方式，当然还有spark-on-yarn等，本文讲述下spark-on-k8s的入门级操作流程，使用的是minikube来搭建单机版的k8s环境，好了废话不多说了，直接如主题：

1、安装minikube环境（本文用到的是1.15.1版本）

进入[https://github.com/kubernetes/minikube/releases/tag/v1.15.1][https_github.com_kubernetes_minikube_releases_tag_v1.15.1]页面下载系统对应的minikube版本，博主用的是ubuntu系统，所以下载

[minikube-linux-amd64][]文件即可

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2pvaG4xMzM3_size_16_color_FFFFFF_t_70][]

2、下载完minikube-linux-amd64后改名为minikube，放在/usr/local/bin目录下，通过chmod +x minikube赋予可执行特性，执行minikube version验证执行结果：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2pvaG4xMzM3_size_16_color_FFFFFF_t_70 1][]

3、下载spark代码：[https://downloads.apache.org/spark/][https_downloads.apache.org_spark]，本文下载的是2.4.7版本

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2pvaG4xMzM3_size_16_color_FFFFFF_t_70 2][]

4、将下载的spark代码解压，进入对应目录（/home/spark/spark-2.4.7-bin-hadoop2.7），在spark主目录执行下面命令：

docker build -t registry.cn-hangzhou.aliyuncs.com/xxx/spark:2.4.7 -f  kubernetes/dockerfiles/spark/Dockerfile  .

5、将打包的spark对象推送到对应的docker仓库

6、使用minikube创建k8s环境：（6G内存，3个CPU），这个配置参考的是spark官网给出的推荐配置[https://spark.apache.org/docs/latest/running-on-kubernetes.html][https_spark.apache.org_docs_latest_running-on-kubernetes.html]

minikube --memory 6144 --cpus 3 start --image-mirror-country=cn --image-repository=registry.cn-hangzhou.aliyuncs.com/google\_containers  --driver=none

当环境创建成功后通过kubectl get po -A获取pods

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2pvaG4xMzM3_size_16_color_FFFFFF_t_70 3][]

7、创建sparkPi运行环境

# 创建spark明明空间
    kubectl create namespace spark
    
    # 创建service账户以及绑定角色
    kubectl create serviceaccount spark-serviceaccount --namespace spark
    kubectl create clusterrolebinding spark-rolebinding --clusterrole=edit --serviceaccount=spark:spark-serviceaccount --namespace=spark

8、在spark目录执行sparkPi脚本：

./bin/spark-submit \
        --master k8s://https://192.168.1.201:8443 \
        --deploy-mode cluster \
        --name spark-pi \
        --class org.apache.spark.examples.SparkPi \
        --conf spark.executor.instances=3 \
        --conf spark.kubernetes.namespace=spark \
        --conf spark.kubernetes.namespace=spark \
        --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark-serviceaccount \
        --conf spark.kubernetes.container.image.pullPolicy=Always \
        --conf spark.kubernetes.container.image=registry.cn-hangzhou.aliyuncs.com/jtl3d/spark:2.4.7 \
        local:///opt/spark/examples/jars/spark-examples_2.11-2.4.7.jar

master配置的地址通过kubectl cluster-info命令获取

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2pvaG4xMzM3_size_16_color_FFFFFF_t_70 4][]

9、查询执行结果，执行命令kubectl get po -A

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2pvaG4xMzM3_size_16_color_FFFFFF_t_70 5][]

查询到spark-pi-driver后执行命令查看执行结果：kubectl logs spark-pi-1606532060711-driver -n spark

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2pvaG4xMzM3_size_16_color_FFFFFF_t_70 6][]

参考文章：

1、[https://gist.github.com/jjstill/8099669931cdfbb90ce6f4c307971514][https_gist.github.com_jjstill_8099669931cdfbb90ce6f4c307971514]

2、[https://spark.apache.org/docs/latest/running-on-kubernetes.html][https_spark.apache.org_docs_latest_running-on-kubernetes.html]

[https_github.com_kubernetes_minikube_releases_tag_v1.15.1]: https://github.com/kubernetes/minikube/releases/tag/v1.15.1
[minikube-linux-amd64]: https://github.com/kubernetes/minikube/releases/download/v1.15.1/minikube-linux-amd64
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2pvaG4xMzM3_size_16_color_FFFFFF_t_70]: /images/20221120/50c53ad3f2b0401d9839b692b81a32fe.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2pvaG4xMzM3_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20201128112528502.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2pvaG4xMzM3,size_16,color_FFFFFF,t_70
[https_downloads.apache.org_spark]: https://downloads.apache.org/spark/
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2pvaG4xMzM3_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/2020112811294373.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2pvaG4xMzM3,size_16,color_FFFFFF,t_70
[https_spark.apache.org_docs_latest_running-on-kubernetes.html]: https://spark.apache.org/docs/latest/running-on-kubernetes.html
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2pvaG4xMzM3_size_16_color_FFFFFF_t_70 3]: https://img-blog.csdnimg.cn/20201128113737255.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2pvaG4xMzM3,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2pvaG4xMzM3_size_16_color_FFFFFF_t_70 4]: https://img-blog.csdnimg.cn/20201128114211852.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2pvaG4xMzM3,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2pvaG4xMzM3_size_16_color_FFFFFF_t_70 5]: https://img-blog.csdnimg.cn/20201128114310925.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2pvaG4xMzM3,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2pvaG4xMzM3_size_16_color_FFFFFF_t_70 6]: https://img-blog.csdnimg.cn/20201128135332787.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2pvaG4xMzM3,size_16,color_FFFFFF,t_70
[https_gist.github.com_jjstill_8099669931cdfbb90ce6f4c307971514]: https://gist.github.com/jjstill/8099669931cdfbb90ce6f4c307971514