使用kubespray部署高可用kubernetes集群

迈不过友情╰ 2022-03-20 02:11 369阅读 0赞

**发布一个k8s部署视频：https://edu.csdn.net/course/detail/26967**

**课程内容：各种k8s部署方式。包括minikube部署，kubeadm部署，kubeasz部署，rancher部署，k3s部署。包括开发测试环境部署k8s，和生产环境部署k8s。**

**腾讯课堂连接地址https://ke.qq.com/course/478827?taid=4373109931462251&tuin=ba64518**

**第二个视频发布  https://edu.csdn.net/course/detail/27109**

**腾讯课堂连接地址https://ke.qq.com/course/484107?tuin=ba64518**

**介绍主要的k8s资源的使用配置和命令。包括configmap，pod，service，replicaset，namespace，deployment，daemonset，ingress，pv，pvc，sc，role，rolebinding，clusterrole，clusterrolebinding，secret，serviceaccount，statefulset，job，cronjob，podDisruptionbudget，podSecurityPolicy,networkPolicy,resourceQuota，limitrange，endpoint，event，conponentstatus，node，apiservice，controllerRevision等。**

**第三个视频发布：https://edu.csdn.net/course/detail/27574**

**详细介绍helm命令，学习helm chart语法，编写helm chart。深入分析各项目源码，学习编写helm插件  
————————————————**

**0 环境**

环境：

<table> 
 <thead> 
  <tr> 
   <th>主机名</th> 
   <th>IP</th> 
  </tr> 
 </thead> 
 <tbody> 
  <tr> 
   <td>k8s-node01</td> 
   <td>172.16.120.151</td> 
  </tr> 
  <tr> 
   <td>k8s-node02</td> 
   <td>172.16.120.152</td> 
  </tr> 
  <tr> 
   <td>k8s-node03</td> 
   <td>172.16.120.153</td> 
  </tr> 
  <tr> 
   <td>ansible-client</td> 
  </tr> 
 </tbody> 
</table>

设置主机名：

1.  `hostnamectl --static set-hostname k8s-node01`
2.  `hostnamectl --static set-hostname k8s-node02`
3.  `hostnamectl --static set-hostname k8s-node03`

### 关闭防火墙（这一步不做也没事，我忘了做了，依然成功） ###

1.  `systemctl stop firewalld`
2.  `systemctl disable firewalld`
3.  `systemctl stop NetworkManager`
4.  `systemctl disable NetworkManager`
5.  sed -i "s/^SELINUX\\=enforcing/SELINUX\\=disabled/g" /etc/selinux/config
6.  setenforce 0

1.  sed -i "s/^SELINUX\\=permissive/SELINUX\\=disabled/g" /etc/selinux/config

# 下载kuberspay源码 #

wget [https://github.com/kubernetes-sigs/kubespray/archive/v2.8.2.tar.gz][https_github.com_kubernetes-sigs_kubespray_archive_v2.8.2.tar.gz]

pip install -r requirements.txt

# vi /etc/hosts #

172.20.164.161 node1
    172.20.164.163 node2
    172.20.164.162 node3
    172.20.164.160 node4

# 2 设置免密登录 #

在ansible-client执行 ssh-keygen -t rsa 生成密钥对

ssh-keygen -t rsa -P ''

将~/.ssh/id\_rsa.pub复制到其他所有节点，这样ansible-client到其他所有节点可以免密登录

1.  `IP=(172.16.120.151 172.16.120.152 172.16.120.153)`
2.  `for x in ${IP[*]}; do ssh-copy-id -i ~/.ssh/id_rsa.pub $x; done`

要使用root权限执行上面操作。

vi inventory/sample/hosts.ini

[all]
    node1 ansible_host=172.20.164.161   ip=172.20.164.161
    node2 ansible_host=172.20.164.163   ip=172.20.164.163
    node3 ansible_host=172.20.164.162   ip=172.20.164.162
    node4 ansible_host=172.20.164.160   ip=172.20.164.160
    
    # ## configure a bastion host if your nodes are not directly reachable
    # bastion ansible_host=x.x.x.x ansible_user=some_user
    
    [kube-master]
    node1
    node2
    
    [etcd]
    node1
    node2
    node3
    
    [kube-node]
    node2
    node3
    node4
    node1
    
    [k8s-cluster:children]
    kube-master
    kube-node

vi inventory/sample/group\_vars/k8s-cluster/k8s-cluster.yml

# Kubernetes configuration dirs and system namespace.
    # Those are where all the additional config stuff goes
    # the kubernetes normally puts in /srv/kubernetes.
    # This puts them in a sane location and namespace.
    # Editing those values will almost surely break something.
    kube_config_dir: /etc/kubernetes
    kube_script_dir: "{
       { bin_dir }}/kubernetes-scripts"
    kube_manifest_dir: "{
       { kube_config_dir }}/manifests"
    
    # This is where all the cert scripts and certs will be located
    kube_cert_dir: "{
       { kube_config_dir }}/ssl"
    
    # This is where all of the bearer tokens will be stored
    kube_token_dir: "{
       { kube_config_dir }}/tokens"
    
    # This is where to save basic auth file
    kube_users_dir: "{
       { kube_config_dir }}/users"
    
    kube_api_anonymous_auth: true
    
    ## Change this to use another Kubernetes version, e.g. a current beta release
    kube_version: v1.12.5
    
    # kubernetes image repo define
    kube_image_repo: "gcr.io/google-containers"
    
    # Where the binaries will be downloaded.
    # Note: ensure that you've enough disk space (about 1G)
    local_release_dir: "/tmp/releases"
    # Random shifts for retrying failed ops like pushing/downloading
    retry_stagger: 5
    
    # This is the group that the cert creation scripts chgrp the
    # cert files to. Not really changeable...
    kube_cert_group: kube-cert
    
    # Cluster Loglevel configuration
    kube_log_level: 2
    
    # Directory where credentials will be stored
    credentials_dir: "{
       { inventory_dir }}/credentials"
    
    # Users to create for basic auth in Kubernetes API via HTTP
    # Optionally add groups for user
    kube_api_pwd: "{
       { lookup('password', credentials_dir + '/kube_user.creds length=15 chars=ascii_letters,digits') }}"
    kube_users:
      kube:
        pass: "{
       {kube_api_pwd}}"
        role: admin
        groups:
          - system:masters
    
    ## It is possible to activate / deactivate selected authentication methods (basic auth, static token auth)
    #kube_oidc_auth: false
    kube_basic_auth: false
    kube_token_auth: false
    
    
    ## Variables for OpenID Connect Configuration https://kubernetes.io/docs/admin/authentication/
    ## To use OpenID you have to deploy additional an OpenID Provider (e.g Dex, Keycloak, ...)
    
    # kube_oidc_url: https:// ...
    # kube_oidc_client_id: kubernetes
    ## Optional settings for OIDC
    # kube_oidc_ca_file: "{
       { kube_cert_dir }}/ca.pem"
    # kube_oidc_username_claim: sub
    # kube_oidc_username_prefix: oidc:
    # kube_oidc_groups_claim: groups
    # kube_oidc_groups_prefix: oidc:
    
    
    # Choose network plugin (cilium, calico, contiv, weave or flannel)
    # Can also be set to 'cloud', which lets the cloud provider setup appropriate routing
    kube_network_plugin: calico
    
    # Kubernetes internal network for services, unused block of space.
    kube_service_addresses: 10.233.0.0/18
    
    # internal network. When used, it will assign IP
    # addresses from this range to individual pods.
    # This network must be unused in your network infrastructure!
    kube_pods_subnet: 10.233.64.0/18
    
    # internal network node size allocation (optional). This is the size allocated
    # to each node on your network.  With these defaults you should have
    # room for 4096 nodes with 254 pods per node.
    kube_network_node_prefix: 24
    
    # The port the API Server will be listening on.
    kube_apiserver_ip: "{
       { kube_service_addresses|ipaddr('net')|ipaddr(1)|ipaddr('address') }}"
    kube_apiserver_port: 6443 # (https)
    kube_apiserver_insecure_port: 8080 # (http)
    # Set to 0 to disable insecure port - Requires RBAC in authorization_modes and kube_api_anonymous_auth: true
    kube_apiserver_insecure_port: 0 # (disabled)
    
    # Kube-proxy proxyMode configuration.
    # Can be ipvs, iptables
    kube_proxy_mode: ipvs
    
    # Kube-proxy nodeport address.
    # cidr to bind nodeport services. Flag --nodeport-addresses on kube-proxy manifest
    kube_proxy_nodeport_addresses: false
    # kube_proxy_nodeport_addresses_cidr: 10.0.1.0/24
    
    ## Encrypting Secret Data at Rest (experimental)
    kube_encrypt_secret_data: false
    
    # DNS configuration.
    # Kubernetes cluster name, also will be used as DNS domain
    cluster_name: cluster.local
    # Subdomains of DNS domain to be resolved via /etc/resolv.conf for hostnet pods
    ndots: 2
    # Can be dnsmasq_kubedns, kubedns, coredns, coredns_dual, manual or none
    dns_mode: coredns
    # Set manual server if using a custom cluster DNS server
    #manual_dns_server: 10.x.x.x
    
    # Can be docker_dns, host_resolvconf or none
    resolvconf_mode: docker_dns
    # Deploy netchecker app to verify DNS resolve as an HTTP service
    deploy_netchecker: false
    # Ip address of the kubernetes skydns service
    skydns_server: "{
       { kube_service_addresses|ipaddr('net')|ipaddr(3)|ipaddr('address') }}"
    skydns_server_secondary: "{
       { kube_service_addresses|ipaddr('net')|ipaddr(4)|ipaddr('address') }}"
    dnsmasq_dns_server: "{
       { kube_service_addresses|ipaddr('net')|ipaddr(2)|ipaddr('address') }}"
    dns_domain: "{
       { cluster_name }}"
    
    ## Container runtime
    ## docker for docker and crio for cri-o.
    container_manager: docker
    
    ## Settings for containerized control plane (etcd/kubelet/secrets)
    etcd_deployment_type: docker
    kubelet_deployment_type: host
    helm_deployment_type: host
    
    # K8s image pull policy (imagePullPolicy)
    k8s_image_pull_policy: IfNotPresent
    
    # audit log for kubernetes
    kubernetes_audit: false
    
    # dynamic kubelet configuration
    dynamic_kubelet_configuration: false
    
    # define kubelet config dir for dynamic kubelet
    #kubelet_config_dir:
    default_kubelet_config_dir: "{
       { kube_config_dir }}/dynamic_kubelet_dir"
    dynamic_kubelet_configuration_dir: "{
       { kubelet_config_dir | default(default_kubelet_config_dir) }}"
    
    # pod security policy (RBAC must be enabled either by having 'RBAC' in authorization_modes or kubeadm enabled)
    podsecuritypolicy_enabled: false
    
    # Make a copy of kubeconfig on the host that runs Ansible in {
       { inventory_dir }}/artifacts
    # kubeconfig_localhost: false
    # Download kubectl onto the host that runs Ansible in {
       { bin_dir }}
    # kubectl_localhost: false
    
    # dnsmasq
    # dnsmasq_upstream_dns_servers:
    #  - /resolvethiszone.with/10.0.4.250
    #  - 8.8.8.8
    
    #  Enable creation of QoS cgroup hierarchy, if true top level QoS and pod cgroups are created. (default true)
    # kubelet_cgroups_per_qos: true
    
    # A comma separated list of levels of node allocatable enforcement to be enforced by kubelet.
    # Acceptable options are 'pods', 'system-reserved', 'kube-reserved' and ''. Default is "".
    # kubelet_enforce_node_allocatable: pods
    
    ## Supplementary addresses that can be added in kubernetes ssl keys.
    ## That can be useful for example to setup a keepalived virtual IP
    # supplementary_addresses_in_ssl_keys: [10.0.0.1, 10.0.0.2, 10.0.0.3]
    
    ## Running on top of openstack vms with cinder enabled may lead to unschedulable pods due to NoVolumeZoneConflict restriction in kube-scheduler.
    ## See https://github.com/kubernetes-sigs/kubespray/issues/2141
    ## Set this variable to true to get rid of this issue
    volume_cross_zone_attachment: false
    # Add Persistent Volumes Storage Class for corresponding cloud provider ( OpenStack is only supported now )
    persistent_volumes_enabled: false
    
    ## Container Engine Acceleration
    ## Enable container acceleration feature, for example use gpu acceleration in containers
    # nvidia_accelerator_enabled: true
    ## Nvidia GPU driver install. Install will by done by a (init) pod running as a daemonset.
    ## Important: if you use Ubuntu then you should set in all.yml 'docker_storage_options: -s overlay2'
    ## Array with nvida_gpu_nodes, leave empty or comment if you dont't want to install drivers.
    ## Labels and taints won't be set to nodes if they are not in the array.
    # nvidia_gpu_nodes:
    #   - kube-gpu-001
    # nvidia_driver_version: "384.111"
    ## flavor can be tesla or gtx
    # nvidia_gpu_flavor: gtx

vi inventory/sample/group\_vars/k8s-cluster/addons.yml

# Kubernetes dashboard
    # RBAC required. see docs/getting-started.md for access details.
    dashboard_enabled: true
    
    # Helm deployment
    helm_enabled: true
    
    # Registry deployment
    registry_enabled: false
    # registry_namespace: kube-system
    # registry_storage_class: ""
    # registry_disk_size: "10Gi"
    
    # Metrics Server deployment
    metrics_server_enabled: true
    # metrics_server_kubelet_insecure_tls: true
    # metrics_server_metric_resolution: 60s
    # metrics_server_kubelet_preferred_address_types: "InternalIP"
    
    # Local volume provisioner deployment
    local_volume_provisioner_enabled: true
    # local_volume_provisioner_namespace: kube-system
    # local_volume_provisioner_storage_classes:
    #   - name: "{
       { local_volume_provisioner_storage_class | default('local-storage') }}"
    #     host_dir: "{
       { local_volume_provisioner_base_dir | default ('/mnt/disks') }}"
    #     mount_dir: "{
       { local_volume_provisioner_mount_dir | default('/mnt/disks') }}"
    #   - name: "local-ssd"
    #     host_dir: "/mnt/local-storage/ssd"
    #     mount_dir: "/mnt/local-storage/ssd"
    #   - name: "local-hdd"
    #     host_dir: "/mnt/local-storage/hdd"
    #     mount_dir: "/mnt/local-storage/hdd"
    #   - name: "local-shared"
    #     host_dir: "/mnt/local-storage/shared"
    #     mount_dir: "/mnt/local-storage/shared"
    
    # CephFS provisioner deployment
    cephfs_provisioner_enabled: false
    # cephfs_provisioner_namespace: "cephfs-provisioner"
    # cephfs_provisioner_cluster: ceph
    # cephfs_provisioner_monitors: "172.24.0.1:6789,172.24.0.2:6789,172.24.0.3:6789"
    # cephfs_provisioner_admin_id: admin
    # cephfs_provisioner_secret: secret
    # cephfs_provisioner_storage_class: cephfs
    # cephfs_provisioner_reclaim_policy: Delete
    # cephfs_provisioner_claim_root: /volumes
    # cephfs_provisioner_deterministic_names: true
    
    # Nginx ingress controller deployment
    ingress_nginx_enabled: true
    ingress_nginx_host_network: true
    # ingress_nginx_nodeselector:
    #   node-role.kubernetes.io/master: ""
    # ingress_nginx_tolerations:
    #   - key: "key"
    #     operator: "Equal"
    #     value: "value"
    #     effect: "NoSchedule"
    # ingress_nginx_namespace: "ingress-nginx"
    ingress_nginx_insecure_port: 80
    ingress_nginx_secure_port: 443
    # ingress_nginx_configmap:
    #   map-hash-bucket-size: "128"
    #   ssl-protocols: "SSLv2"
    # ingress_nginx_configmap_tcp_services:
    #   9000: "default/example-go:8080"
    # ingress_nginx_configmap_udp_services:
    #   53: "kube-system/kube-dns:53"
    
    # Cert manager deployment
    cert_manager_enabled: true
    cert_manager_namespace: "cert-manager"

# 安装集群 #

ansible-playbook -i inventory/sample/hosts.ini cluster.yml -b -v --private-key=~/.ssh/id_rsa

如果要改云提供者可以改cat inventory/sample/group\_vars/all/all.yml这个文件

[https_github.com_kubernetes-sigs_kubespray_archive_v2.8.2.tar.gz]: https://github.com/kubernetes-sigs/kubespray/archive/v2.8.2.tar.gz