aws--efs---使用efs创建k8s的动态pvc

╰+哭是因爲堅強的太久メ 2023-07-18 06:17 1阅读 0赞

我们之前使用EBS作为k8s的动态pvc的pv，但是EBS不支持ReadWriteMany类型的声明。

所以尝试使用efs作为存储底层来声明pvc。

# 动态pvc声明方案 #

## 创建efs文件存储系统 ##

使用命令

# 创建EFS Security group
    VPC_ID=$(aws eks describe-cluster --name ${CLUSTER_NAME} --region ${AWS_REGION} --query "cluster.resourcesVpcConfig.vpcId" --output text)
    VPC_CIDR=$(aws ec2 describe-vpcs --vpc-ids ${VPC_ID} --query "Vpcs[].CidrBlock"  --region ${AWS_REGION} --output text)
    aws ec2 create-security-group --description ${CLUSTER_NAME}-efs-eks-sg --group-name efs-sg --vpc-id ${VPC_ID}
    SGGroupID=上一步的结果访问
    aws ec2 authorize-security-group-ingress --group-id ${SGGroupID}  --protocol tcp --port 2049 --cidr ${VPC_CIDR}
    
    # 创建EFS file system 和 mount-target, 请根据你的环境替换 FileSystemId， SubnetID， SGGroupID
    aws efs create-file-system --creation-token eks-efs --region ${AWS_REGION}
    aws efs create-mount-target --file-system-id FileSystemId --subnet-id SubnetID --security-group SGGroupID

也可以通过界面操作，步骤参考

[利用 EFS 快速搭建 NFS 文件系统][EFS _ NFS]

## 安装efs驱动csi项目 ##

这里需要借助 Amazon EFS CSI Driver项目，[csi项目github地址][csi_github]

git clone https://github.com/kubernetes-sigs/aws-efs-csi-driver
    
    cd aws-efs-csi-driver-master/deploy/kubernetes/overlays/stable
    
    cat kustomization.yaml

当前版本内容如下:

apiVersion: kustomize.config.k8s.io/v1beta1
    kind: Kustomization
    bases:
    - ../../base
    images:
    - name: amazon/aws-efs-csi-driver
      newName: 602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/aws-efs-csi-driver
      newTag: v0.2.0
    - name: quay.io/k8scsi/livenessprobe
      newName: 602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/csi-liveness-probe
      newTag: v1.1.0
    - name: quay.io/k8scsi/csi-node-driver-registrar
      newName: 602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/csi-node-driver-registrar
      newTag: v1.1.0

需要修改如下:

apiVersion: kustomize.config.k8s.io/v1beta1
    kind: Kustomization
    bases:
    - ../../base
    images:
    - name: amazon/aws-efs-csi-driver
      newTag: v0.3.0
      newName: amazon/aws-efs-csi-driver
    - name: quay.io/k8scsi/livenessprobe
      newName: 602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/csi-liveness-probe
      newTag: v1.1.0
    - name: quay.io/k8scsi/csi-node-driver-registrar
      newName: 602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/csi-node-driver-registrar
      newTag: v1.1.0

创建

kubectl create -k .
    
    kubectl get pods -n kube-system
    
    NAME                                      READY   STATUS    RESTARTS   AGE
    alb-ingress-controller-649b854d75-m8c75   1/1     Running   0          2d18h
    aws-node-ct6rz                            1/1     Running   0          4d18h
    aws-node-sfjtn                            1/1     Running   0          3d21h
    aws-node-xzfx9                            1/1     Running   0          4d18h
    coredns-6565755d58-pd5nm                  1/1     Running   0          4d18h
    coredns-6565755d58-v9nl7                  1/1     Running   0          4d18h
    ebs-csi-controller-6dcc4dc6f4-6k4s5       4/4     Running   0          2d17h
    ebs-csi-controller-6dcc4dc6f4-vtklz       4/4     Running   0          2d17h
    ebs-csi-node-2zmct                        3/3     Running   0          2d17h
    ebs-csi-node-plljf                        3/3     Running   0          2d17h
    ebs-csi-node-s9lbz                        3/3     Running   0          2d17h
    efs-csi-node-5jtlc                        3/3     Running   0          10h
    efs-csi-node-lqdz9                        3/3     Running   0          10h
    efs-csi-node-snqmh                        3/3     Running   0          10h
    kube-proxy-g4mcw                          1/1     Running   0          4d18h
    kube-proxy-mb88w                          1/1     Running   0          4d18h
    kube-proxy-tpx4x                          1/1     Running   0          3d21h
    kubernetes-dashboard-5f7b999d65-dcc6h     1/1     Running   0          2d23h
    metrics-server-7fcf9cc98b-rntrh           1/1     Running   0          44h
    
    
    kubectl exec -ti efs-csi-node-5jtlc -n kube-system -- mount.efs --version
    # Make sure the version is > 1.19

## 可能遇到的问题—Failed to resolve “fs-XXXXX.efs.cn-northwest-1.amazonaws.com” - check that your file system ID is correct ##

我们在创建efs的界面中其实可以看到 中国区的DNS是带有cn后缀的，比如:amazonaws.com.cn

eks-workshop-greater-china的文档中说v0.3.0不支持中国区，需要用v0.2.0。

https://github.com/kubernetes-sigs/aws-efs-csi-driver/issues/138 v0.2.0 image contains old version of efs-utils, efs-utils added China region support from v1.19 The v.0.3.0 does work, you can also build your image to use v.0.2.0 CSI
    
    v0.2.0版本的镜像中包含了老版本的efs-utils从v1.19增加对中国区的支持,但是v0.3.0是不起作用的，需要使用v0.2.0版本的镜像。

这个问题其实v0.3.0已经修复了，而且v0.2.0反而不支持中国区，目前下载的master的kustomization.yaml默认使用的就是v0.2.0，是会遇到问题的，无法转换出 带有cn后缀的 DNS路径，所以需要使用v0.3.0。

如上文使用v0.3.0的镜像即可

详情参考链接:  
[support EFS in china regions][]

解决方案  
as v0.3.0 is released: https://github.com/kubernetes-sigs/aws-efs-csi-driver/releases/tag/v0.3.0  
Please use amazon/aws-efs-csi-driver:v0.3.0

[The endpoints in China region are different from others][]

## 部署样例测试 ##

## Deploy app use the EFS
    cd examples/kubernetes/multiple_pods/
    aws efs describe-file-systems --query "FileSystems[*].[FileSystemId,Name]" --region ${AWS_REGION} --output text
    
    # 修改 the specs/pv.yaml file and replace the volumeHandle with FILE_SYSTEM_ID
    # 例子：
    #csi:
    #    driver: efs.csi.aws.com
    #    volumeHandle: fs-9c21a999
    
    
    # 部署 the efs-sc storage class, efs-claim pv claim, efs-pv, and app1 and app2 sample applications.
    kubectl apply -f specs/
    
    kubectl describe storageclass efs-sc
    kubectl get pv
    kubectl describe pv efs-pv
    kubectl get pods --watch
    kubectl get events
    
    # 验证
    kubectl exec -ti app1 -- tail /data/out1.txt
    kubectl exec -ti app2 -- tail /data/out1.txt
    
    # 清理
    kubectl delete -f specs/

# 静态pvc声明方案 #

pv.yaml

apiVersion: v1
    kind: PersistentVolume
    metadata:
      name: pv0001
    spec:
      capacity:
        storage: 1Gi
      accessModes:
      - ReadWriteOnce
      nfs:
        path: /tmp
        server: 172.17.0.2
      persistentVolumeReclaimPolicy: Retain            #Here is policy
      claimRef:                                        #Here is claim reference
        name: claim1
        namespace: default

pvc.yaml

apiVersion: "v1"
    kind: "PersistentVolumeClaim"
    metadata:
      name: "claim1"
    spec:
      accessModes:
        - "ReadWriteOnce"
      resources:
        requests:
          storage: "1Gi"
      volumeName: "pv0001"

# 参考资料 #

[步骤6 配置使用EFS][6 _EFS]

[How do I use Amazon EFS with Amazon EKS][]

[eks+ efs 在 Amazon EKS 上使用 Kubeflow 进行分布式 TensorFlow 训练][eks_ efs _ Amazon EKS _ Kubeflow _ TensorFlow]

[Persistent Volumes][]

[pv and pvc - kube- efs][]

[EFS _ NFS]: https://blog.csdn.net/zzq900503/article/details/105074948
[csi_github]: https://github.com/kubernetes-sigs/aws-efs-csi-driver
[support EFS in china regions]: https://github.com/kubernetes-sigs/aws-efs-csi-driver/issues/138
[The endpoints in China region are different from others]: https://github.com/kubernetes-sigs/aws-efs-csi-driver/issues/142
[6 _EFS]: https://github.com/aws-samples/eks-workshop-greater-china/blob/master/china/2020_EKS_Launch_Workshop/%E6%AD%A5%E9%AA%A46-%E9%85%8D%E7%BD%AE%E4%BD%BF%E7%94%A8EFS.md?from=groupmessage
[How do I use Amazon EFS with Amazon EKS]: https://aws.amazon.com/tw/premiumsupport/knowledge-center/eks-pods-efs/
[eks_ efs _ Amazon EKS _ Kubeflow _ TensorFlow]: https://aws.amazon.com/cn/blogs/china/distributed-tensorflow-training-using-kubeflow-on-amazon-eks/
[Persistent Volumes]: https://kubernetes.io/docs/concepts/storage/persistent-volumes/
[pv and pvc - kube- efs]: https://serverfault.com/questions/933673/pv-and-pvc-kube-efs