helm部署jenkins到k8s并创建pipeline构建项目

Dear 丶 2022-11-21 01:06 610阅读 0赞

# 安装部署nfs #

1. nfs服务器创建目录

[yeqiang@harbor ~]$ sudo mkdir /home/nfs/jenkins -p

2. 设置其他用户可以写入该目录

[yeqiang@harbor ~]$ sudo chmod o+w /home/nfs/jenkins/ -R

3. 配置/etc/exports

/home/nfs/jenkins 192.168.1.167(rw,sync,no_subtree_check,no_root_squash)
    /home/nfs/jenkins/agent 192.168.1.167(rw,sync,no_subtree_check,no_root_squash)

4. 启动nfs服务，配置开启自启动

[yeqiang@harbor ~]$ sudo systemctl enable nfs-server --now
    Created symlink /etc/systemd/system/multi-user.target.wants/nfs-server.service → /usr/lib/systemd/system/nfs-server.service.

5. 检查服务及端口(111/tcp)状态

[yeqiang@harbor ~]$ netstat -lnetp | grep 111
    (Not all processes could be identified, non-owned process info
     will not be shown, you would have to be root to see it all.)
    tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      0          19646      -                   
    tcp6       0      0 :::111                  :::*                    LISTEN      0          19650      -                   
    [yeqiang@harbor ~]$ sudo exportfs -v
    /home/nfs/jenkins
    		192.168.1.167(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash)
    /home/nfs/jenkins/agent
    		192.168.1.167(sync,wdelay,hide,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash)

# 创建k8s存储 #

1. 创建StorageClass，配置文件nfs-storege-class.yml

kind: StorageClass
    apiVersion: storage.k8s.io/v1
    metadata:
      name: nfs-storage
      annotations:
        storageclass.kubernetes.io/is-default-class: 'false'
    provisioner: kubernetes.io/no-provisioner
    reclaimPolicy: Delete
    volumeBindingMode: WaitForFirstConsumer

执行创建

[yeqiang@harbor k8s]$ kubectl apply -f nfs-storege-class.yml 
    storageclass.storage.k8s.io/nfs-storage created
    [yeqiang@harbor k8s]$ kubectl get storageclass
    NAME                      PROVISIONER                    RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
    nfs-storage               kubernetes.io/no-provisioner   Delete          WaitForFirstConsumer   false                  6s

2. 创建PersistantVolume，配置文件jenkins-pv.yml

apiVersion: v1
    kind: PersistentVolume
    metadata:
      name: jenkins-pv
    spec:
      capacity:
        storage: 10Gi
      accessModes:
        - ReadWriteOnce
      volumeMode: Filesystem
      persistentVolumeReclaimPolicy: Recycle
      storageClassName: "nfs-storage"
      nfs:
        # 根据实际共享目录修改
        path: /home/nfs/jenkins
        # 根据实际 nfs服务器地址修改
        server: 192.168.1.167

执行创建

[yeqiang@harbor k8s]$ kubectl apply -f jenkins-pv.yml 
    persistentvolume/jenkins-pv created
    [yeqiang@harbor k8s]$ kubectl get pv 
    NAME                          CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                               STORAGECLASS    REASON   AGE
    jenkins-pv                    10Gi       RWO            Recycle          Available                                       nfs-storage              6s

# helm部署jenkins #

1. helm查看jenkins信息

[yeqiang@harbor k8s]$ helm repo add jenkins https://charts.jenkins.io
    "jenkins" has been added to your repositories
    [yeqiang@harbor k8s]$ helm search repo jenkins
    NAME           	CHART VERSION	APP VERSION	DESCRIPTION                                       
    jenkins/jenkins	2.15.1       	lts        	Open source continuous integration server. It s...
    stable/jenkins 	2.5.4        	lts        	DEPRECATED - Open source continuous integration...

**注意stable/jenkins已经标注过时了，我们选择jenkins/jenkins**

2. 创建namespace

[yeqiang@harbor k8s]$ kubectl create ns jenkins
    namespace/jenkins created

3. helm安装

[yeqiang@harbor k8s]$ helm install jenkins jenkins/jenkins -n jenkins --set persistence.storageClass="nfs-storage"
    NAME: jenkins
    LAST DEPLOYED: Thu Oct 29 10:25:54 2020
    NAMESPACE: jenkins
    STATUS: deployed
    REVISION: 1
    NOTES:
    1. Get your 'admin' user password by running:
      printf $(kubectl get secret --namespace jenkins jenkins -o jsonpath="{.data.jenkins-admin-password}" | base64 --decode);echo
    2. Get the Jenkins URL to visit by running these commands in the same shell:
      export POD_NAME=$(kubectl get pods --namespace jenkins -l "app.kubernetes.io/component=jenkins-master" -l "app.kubernetes.io/instance=jenkins" -o jsonpath="{.items[0].metadata.name}")
      echo http://127.0.0.1:8080
      kubectl --namespace jenkins port-forward $POD_NAME 8080:8080
    
    3. Login with the password from step 1 and the username: admin
    
    4. Use Jenkins Configuration as Code by specifying configScripts in your values.yaml file, see documentation: http:///configuration-as-code and examples: https://github.com/jenkinsci/configuration-as-code-plugin/tree/master/demos
    
    For more information on running Jenkins on Kubernetes, visit:
    https://cloud.google.com/solutions/jenkins-on-container-engine
    For more information about Jenkins Configuration as Code, visit:
    https://jenkins.io/projects/jcasc/

参数说明：

persistence.storageClass 赋值为刚才创建的StorageClass名称nfs-sotrage

4. 故障排除

[yeqiang@harbor k8s]$ kubectl get pods -n jenkins
    NAME                       READY   STATUS    RESTARTS   AGE
    jenkins-58978c6d6d-dszqz   1/2     Running   2          6m8s
    [yeqiang@harbor k8s]$ kubectl describe pod jenkins-58978c6d6d-dszqz -n jenkins
    Name:         jenkins-58978c6d6d-dszqz
    Namespace:    jenkins
    Priority:     0
    Node:         192.168.1.167/192.168.1.167
    Start Time:   Thu, 29 Oct 2020 10:25:55 +0800
    Labels:       app.kubernetes.io/component=jenkins-master
                  app.kubernetes.io/instance=jenkins
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=jenkins
                  pod-template-hash=58978c6d6d
    Annotations:  checksum/config: 9823a20535eb4e992b6741805b204c7c82d33080079887578e27c24328df7b29
    Status:       Running
    IP:           172.20.0.108
    IPs:
      IP:           172.20.0.108
    Controlled By:  ReplicaSet/jenkins-58978c6d6d
    Init Containers:
      copy-default-config:
        Container ID:  docker://e29d8d6285ddebaf565a206213da47aac723ae79a883ff6108aafd7108a5b1fd
        Image:         jenkins/jenkins:lts
        Image ID:      docker-pullable://jenkins/jenkins@sha256:aa1ec2a6a106608a286dc06c334053fced1df87a7cb307853ca1cf8fa48ba09b
        Port:          <none>
        Host Port:     <none>
        Command:
          sh
          /var/jenkins_config/apply_config.sh
        State:          Terminated
          Reason:       Completed
          Exit Code:    0
          Started:      Thu, 29 Oct 2020 10:25:58 +0800
          Finished:     Thu, 29 Oct 2020 10:26:40 +0800
        Ready:          True
        Restart Count:  0
        Limits:
          cpu:     2
          memory:  4Gi
        Requests:
          cpu:     50m
          memory:  256Mi
        Environment:
          ADMIN_PASSWORD:  <set to the key 'jenkins-admin-password' in secret 'jenkins'>  Optional: false
          ADMIN_USER:      <set to the key 'jenkins-admin-user' in secret 'jenkins'>      Optional: false
        Mounts:
          /tmp from tmp (rw)
          /usr/share/jenkins/ref/plugins from plugins (rw)
          /var/jenkins_config from jenkins-config (rw)
          /var/jenkins_home from jenkins-home (rw)
          /var/jenkins_plugins from plugin-dir (rw)
          /var/run/secrets/kubernetes.io/serviceaccount from jenkins-token-krt5l (ro)
    Containers:
      jenkins:
        Container ID:  docker://13cb74bdc25782b5bf7258fc4c369544571866c3a029dbc26f9ae489fb7b742b
        Image:         jenkins/jenkins:lts
        Image ID:      docker-pullable://jenkins/jenkins@sha256:aa1ec2a6a106608a286dc06c334053fced1df87a7cb307853ca1cf8fa48ba09b
        Ports:         8080/TCP, 50000/TCP
        Host Ports:    0/TCP, 0/TCP
        Args:
          --argumentsRealm.passwd.$(ADMIN_USER)=$(ADMIN_PASSWORD)
          --argumentsRealm.roles.$(ADMIN_USER)=admin
          --httpPort=8080
        State:          Running
          Started:      Thu, 29 Oct 2020 10:31:24 +0800
        Last State:     Terminated
          Reason:       Error
          Exit Code:    143
          Started:      Thu, 29 Oct 2020 10:29:01 +0800
          Finished:     Thu, 29 Oct 2020 10:31:18 +0800
        Ready:          False
        Restart Count:  2
        Limits:
          cpu:     2
          memory:  4Gi
        Requests:
          cpu:      50m
          memory:   256Mi
        Liveness:   http-get http://:http/login delay=90s timeout=5s period=10s #success=1 #failure=5
        Readiness:  http-get http://:http/login delay=60s timeout=5s period=10s #success=1 #failure=3
        Environment:
          POD_NAME:                  jenkins-58978c6d6d-dszqz (v1:metadata.name)
          JAVA_OPTS:                 -Dcasc.reload.token=$(POD_NAME)
          JENKINS_OPTS:              
          JENKINS_SLAVE_AGENT_PORT:  50000
          ADMIN_PASSWORD:            <set to the key 'jenkins-admin-password' in secret 'jenkins'>  Optional: false
          ADMIN_USER:                <set to the key 'jenkins-admin-user' in secret 'jenkins'>      Optional: false
          CASC_JENKINS_CONFIG:       /var/jenkins_home/casc_configs
        Mounts:
          /tmp from tmp (rw)
          /usr/share/jenkins/ref/plugins/ from plugin-dir (rw)
          /var/jenkins_config from jenkins-config (ro)
          /var/jenkins_home from jenkins-home (rw)
          /var/jenkins_home/casc_configs from sc-config-volume (rw)
          /var/run/secrets/kubernetes.io/serviceaccount from jenkins-token-krt5l (ro)
      jenkins-sc-config:
        Container ID:   docker://a33d84511d702b29e1d1b206793666bb360f071e0f7577a24f660c8a0d4eef1b
        Image:          kiwigrid/k8s-sidecar:0.1.193
        Image ID:       docker-pullable://kiwigrid/k8s-sidecar@sha256:170069ff097679f8fafcf01510d1fe05260af683b949fd2b05866e18ca4d9c74
        Port:           <none>
        Host Port:      <none>
        State:          Running
          Started:      Thu, 29 Oct 2020 10:26:48 +0800
        Ready:          True
        Restart Count:  0
        Environment:
          POD_NAME:           jenkins-58978c6d6d-dszqz (v1:metadata.name)
          LABEL:              jenkins-jenkins-config
          FOLDER:             /var/jenkins_home/casc_configs
          NAMESPACE:          jenkins
          REQ_URL:            http://localhost:8080/reload-configuration-as-code/?casc-reload-token=$(POD_NAME)
          REQ_METHOD:         POST
          REQ_RETRY_CONNECT:  10
        Mounts:
          /var/jenkins_home from jenkins-home (rw)
          /var/jenkins_home/casc_configs from sc-config-volume (rw)
          /var/run/secrets/kubernetes.io/serviceaccount from jenkins-token-krt5l (ro)
    Conditions:
      Type              Status
      Initialized       True 
      Ready             False 
      ContainersReady   False 
      PodScheduled      True 
    Volumes:
      plugins:
        Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
        Medium:     
        SizeLimit:  <unset>
      tmp:
        Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
        Medium:     
        SizeLimit:  <unset>
      jenkins-config:
        Type:      ConfigMap (a volume populated by a ConfigMap)
        Name:      jenkins
        Optional:  false
      plugin-dir:
        Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
        Medium:     
        SizeLimit:  <unset>
      jenkins-home:
        Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
        ClaimName:  jenkins
        ReadOnly:   false
      sc-config-volume:
        Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
        Medium:     
        SizeLimit:  <unset>
      jenkins-token-krt5l:
        Type:        Secret (a volume populated by a Secret)
        SecretName:  jenkins-token-krt5l
        Optional:    false
    QoS Class:       Burstable
    Node-Selectors:  <none>
    Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                     node.kubernetes.io/unreachable:NoExecute for 300s
    Events:
      Type     Reason     Age                    From                   Message
      ----     ------     ----                   ----                   -------
      Normal   Scheduled  6m10s                  default-scheduler      Successfully assigned jenkins/jenkins-58978c6d6d-dszqz to 192.168.1.167
      Normal   Pulling    6m9s                   kubelet, 192.168.1.167  Pulling image "jenkins/jenkins:lts"
      Normal   Pulled     6m8s                   kubelet, 192.168.1.167  Successfully pulled image "jenkins/jenkins:lts"
      Normal   Created    6m7s                   kubelet, 192.168.1.167  Created container copy-default-config
      Normal   Started    6m7s                   kubelet, 192.168.1.167  Started container copy-default-config
      Normal   Pulled     5m18s                  kubelet, 192.168.1.167  Successfully pulled image "jenkins/jenkins:lts"
      Normal   Created    5m18s                  kubelet, 192.168.1.167  Created container jenkins
      Normal   Started    5m17s                  kubelet, 192.168.1.167  Started container jenkins
      Normal   Pulled     5m17s                  kubelet, 192.168.1.167  Container image "kiwigrid/k8s-sidecar:0.1.193" already present on machine
      Normal   Created    5m17s                  kubelet, 192.168.1.167  Created container jenkins-sc-config
      Normal   Started    5m17s                  kubelet, 192.168.1.167  Started container jenkins-sc-config
      Warning  Unhealthy  3m11s (x7 over 4m11s)  kubelet, 192.168.1.167  Readiness probe failed: Get http://172.20.0.108:8080/login: dial tcp 172.20.0.108:8080: connect: connection refused
      Normal   Pulling    3m7s (x2 over 5m24s)   kubelet, 192.168.1.167  Pulling image "jenkins/jenkins:lts"
      Normal   Killing    3m7s                   kubelet, 192.168.1.167  Container jenkins failed liveness probe, will be restarted
      Warning  Unhealthy  67s (x8 over 3m47s)    kubelet, 192.168.1.167  Liveness probe failed: Get http://172.20.0.108:8080/login: dial tcp 172.20.0.108:8080: connect: connection refused

Liveness probe failed：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hrbmFydXRv_size_16_color_FFFFFF_t_70][]

把deployments里面的jenkins配置项目failureThreshold改为10

原因：jenkins默认引用的国外更新源，初始化时间较长，pod还未就绪就被k8s杀死了。

可能应为网络问题，pod可能还会反创建。大约十几分钟后。可以正常访问

5. 访问Jenkins

查看访问地址

[yeqiang@harbor k8s]$ kubectl get svc -n jenkins
    NAME            TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)     AGE
    jenkins         ClusterIP   10.68.62.85   <none>        8080/TCP    26m
    jenkins-agent   ClusterIP   10.68.46.25   <none>        50000/TCP   26m

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hrbmFydXRv_size_16_color_FFFFFF_t_70 1][]

查看登陆用户及密码

[yeqiang@harbor k8s]$ kubectl edit secret jenkins -n jenkins

# Please edit the object below. Lines beginning with a '#' will be ignored,
    # and an empty file will abort the edit. If an error occurs while saving this file will be
    # reopened with the relevant failures.
    #
    apiVersion: v1
    data:
      jenkins-admin-password: clM5bXgxMUNvQw==
      jenkins-admin-user: YWRtaW4=
    kind: Secret
    metadata:
      annotations:
        meta.helm.sh/release-name: jenkins
        meta.helm.sh/release-namespace: jenkins
      creationTimestamp: "2020-10-29T02:25:54Z"
      labels:
        app.kubernetes.io/component: jenkins-master
        app.kubernetes.io/instance: jenkins
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: jenkins
        helm.sh/chart: jenkins-2.15.1
      managedFields:
      - apiVersion: v1
        fieldsType: FieldsV1
        fieldsV1:
          f:data:
            .: {}
            f:jenkins-admin-password: {}
            f:jenkins-admin-user: {}
          f:metadata:
            f:annotations:
              .: {}
              f:meta.helm.sh/release-name: {}
              f:meta.helm.sh/release-namespace: {}
            f:labels:
              .: {}
              f:app.kubernetes.io/component: {}
              f:app.kubernetes.io/instance: {}
              f:app.kubernetes.io/managed-by: {}
              f:app.kubernetes.io/name: {}
              f:helm.sh/chart: {}
          f:type: {}
        manager: Go-http-client
        operation: Update
        time: "2020-10-29T02:25:54Z"
      name: jenkins
      namespace: jenkins
      resourceVersion: "2557784"
      selfLink: /api/v1/namespaces/jenkins/secrets/jenkins
      uid: e8eb14b7-2e7c-42b4-83b5-39889042e72b
    type: Opaque

[yeqiang@harbor k8s]$ echo 'YWRtaW4=' | base64 -d
    admin
    [yeqiang@harbor k8s]$ echo 'clM5bXgxMUNvQw==' | base64 -d
    rS9mx11CoC

得到用户admin，密码rS9mx11CoC

登陆系统

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hrbmFydXRv_size_16_color_FFFFFF_t_70 2][]

# 创建jenkins PipeLine构建部署项目 #

1. 查看jenkins cloud 配置

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hrbmFydXRv_size_16_color_FFFFFF_t_70 3][]

2. 编写pipeline

def label = "jenkins-slave"
    podTemplate(
        label: label, 
        cloud: 'kubernetes', 
        containers: [containerTemplate(alwaysPullImage: true, args: 'cat', command: '/bin/sh -c', image: 'maven:3.6.3-openjdk-8', livenessProbe: containerLivenessProbe(execArgs: '', failureThreshold: 0, initialDelaySeconds: 0, periodSeconds: 0, successThreshold: 0, timeoutSeconds: 0), name: 'maven', resourceLimitCpu: '', resourceLimitMemory: '', resourceRequestCpu: '', resourceRequestMemory: '', ttyEnabled: true, workingDir: '/home/jenkins/agent')],     
        workspaceVolume: nfsWorkspaceVolume(serverAddress: '192.168.1.167', serverPath: '/home/nfs/jenkins/agent', readOnly: 'false')
    ) {
        node(label) {
            stage('git') {
                git 'https://github.com/hknarutofk/spring-demo.git'
            }
            
            stage('mvn'){
    	    container('maven'){	        
    		sh '''
    cat > /usr/share/maven/conf/settings.xml << EOF
    <?xml version="1.0" encoding="UTF-8"?>
    <settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
    	  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    	  xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">
      <localRepository>/root/.m2</localRepository>
      <mirrors>
        <mirror>
          <id>alimaven</id>
          <name>aliyun maven</name>
          <url>http://maven.aliyun.com/nexus/content/groups/public/</url>
          <mirrorOf>central</mirrorOf>        
        </mirror>
      </mirrors>
    </settings>
    EOF
    '''		
            
    		sh "mvn clean package -Dmaven.test.skip=true"
    	    }            
            }
        }
    }

故障排除：

Could not resolve host

放弃使用镜像maven:3.3.9-jdk-8-alpine，采用maven:3.6.3-openjdk-8。原因：maven:3.3.9-jdk-8-alpine对/etc/resolv.conf解析有故障

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hrbmFydXRv_size_16_color_FFFFFF_t_70]: /images/20221120/a88fa56e6ce34c23b32c7842d0693db1.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hrbmFydXRv_size_16_color_FFFFFF_t_70 1]: /images/20221120/a296eedd33a447d490b8669196cd5dcf.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hrbmFydXRv_size_16_color_FFFFFF_t_70 2]: /images/20221120/d4ede77975ea4d3a8d0bc052c08d25e5.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hrbmFydXRv_size_16_color_FFFFFF_t_70 3]: https://img-blog.csdnimg.cn/20201029140813812.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hrbmFydXRv,size_16,color_FFFFFF,t_70