[K8S] Service代理模式之IPVS

小咪咪 2022-09-07 09:28 167阅读 0赞

相比于iptables(由于规则遍历,随着访问量增加,匹配变慢), IPVS(使用哈希表)性能更佳

1.Service默认使用的代理模式是iptables,可以查看kube-proxy pod验证

[root@k8s-master ~]# kubectl get pod -n kube-system
    NAME                                      READY   STATUS    RESTARTS   AGE
    calico-kube-controllers-97769f7c7-xh7j7   1/1     Running   18         27d
    calico-node-58ck4                         1/1     Running   20         27d
    calico-node-q6qxm                         1/1     Running   17         27d
    calico-node-vmmv5                         1/1     Running   17         27d
    coredns-7f89b7bc75-mgtnj                  1/1     Running   17         27d
    coredns-7f89b7bc75-wkrjq                  1/1     Running   17         27d
    etcd-k8s-master                           1/1     Running   18         27d
    kube-apiserver-k8s-master                 1/1     Running   21         27d
    kube-controller-manager-k8s-master        1/1     Running   14         15d
    kube-proxy-clmzc                          1/1     Running   17         27d
    kube-proxy-f4mk4                          1/1     Running   17         27d
    kube-proxy-s7n6w                          1/1     Running   18         27d
    kube-scheduler-k8s-master                 1/1     Running   14         15d
    metrics-server-84f9866fdf-9l6rt           1/1     Running   16         16d
    [root@k8s-master ~]#

没有指定mode,默认使用iptables

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d5X2hoeHg_size_16_color_FFFFFF_t_70][]

2.切换Sevice模式为ipvs

kubectl edit configmap kube-proxy -n kube-system 编辑configmap将mode改为ipvs

![20210822222512669.png][]

需要重启pod, 以 kube-proxy-clmzc为例

[root@k8s-master ~]# kubectl get pod -o wide -n kube-system | grep kube-proxy
    kube-proxy-clmzc                          1/1     Running   17         27d   192.168.231.122   k8s-node1    <none>           <none>
    kube-proxy-f4mk4                          1/1     Running   17         27d   192.168.231.123   k8s-node2    <none>           <none>
    kube-proxy-s7n6w                          1/1     Running   18         27d   192.168.231.121   k8s-master   <none>           <none>
    [root@k8s-master ~]#
    [root@k8s-master ~]# kubectl delete pod kube-proxy-clmzc -n kube-system
    pod "kube-proxy-clmzc" deleted
    [root@k8s-master ~]# kubectl get pod -o wide -n kube-system | grep kube-proxy
    kube-proxy-f4mk4                          1/1     Running   17         27d   192.168.231.123   k8s-node2    <none>           <none>
    kube-proxy-frvx7                          1/1     Running   0          19s   192.168.231.122   k8s-node1    <none>           <none>
    kube-proxy-s7n6w                          1/1     Running   18         27d   192.168.231.121   k8s-master   <none>           <none>
    [root@k8s-master ~]#

3.使用ipvsadm -L -n验证

重启的pod在 192.168.231.122, 即k8s-node1,如果没有ipvsadm先安装, yum install ipvsadm -y

svc的cluster IP 是10.104.247.10, nodePort 是31947  
endpoints 是10.244.169.161:80,10.244.169.163:80,10.244.169.165:80

[root@k8s-master ~]# kubectl get svc
    NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE
    kubernetes   ClusterIP   10.96.0.1       <none>        443/TCP        27d
    web          NodePort    10.104.247.10   <none>        80:31947/TCP   101m
    [root@k8s-master ~]#
    [root@k8s-master ~]# kubectl get ep
    NAME         ENDPOINTS                                               AGE
    kubernetes   192.168.231.121:6443                                    27d
    web          10.244.169.161:80,10.244.169.163:80,10.244.169.165:80   102m
    [root@k8s-master ~]#

在k8s-node1执行ipvsadm -L -n,可以找到集群内部/外部转发,调度算法是默认的rr

[root@k8s-node1 ~]# ipvsadm -L -n
    IP Virtual Server version 1.2.1 (size=4096)
    Prot LocalAddress:Port Scheduler Flags
      -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
    ...
    TCP  10.104.247.10:80 rr
      -> 10.244.169.161:80            Masq    1      0          0
      -> 10.244.169.163:80            Masq    1      0          0
      -> 10.244.169.165:80            Masq    1      0          0
    ...
    TCP  192.168.122.1:31947 rr
      -> 10.244.169.161:80            Masq    1      0          0
      -> 10.244.169.163:80            Masq    1      0          0
      -> 10.244.169.165:80            Masq    1      0          0
    UDP  10.96.0.10:53 rr
      -> 10.244.169.149:53            Masq    1      0          0
      -> 10.244.169.156:53            Masq    1      0          0
    [root@k8s-node1 ~]#

说明：一些调度算法  
rr: round-robin  
lc: least connection  
dh: destination hashing  
sh: source hashing  
sed: shortest expected delay  
nq: never queue  
……

4.恢复环境

kubectl edit configmap kube-proxy -n kube-system 编辑configmap

![20210822225509824.png][]

重启k8s-node1上 kube-proxy的pod

[root@k8s-master ~]# kubectl edit configmap kube-proxy -n kube-system
    configmap/kube-proxy edited
    [root@k8s-master ~]#
    [root@k8s-master ~]# kubectl get pod -o wide -n kube-system | grep kube-proxy
    kube-proxy-f4mk4                          1/1     Running   17         27d   192.168.231.123   k8s-node2    <none>           <none>
    kube-proxy-frvx7                          1/1     Running   0          26m   192.168.231.122   k8s-node1    <none>           <none>
    kube-proxy-s7n6w                          1/1     Running   18         27d   192.168.231.121   k8s-master   <none>           <none>
    [root@k8s-master ~]# kubectl delete pod kube-proxy-frvx7 -n kube-system
    pod "kube-proxy-frvx7" deleted
    [root@k8s-master ~]# kubectl get pod -o wide -n kube-system | grep kube-proxy
    kube-proxy-f4mk4                          1/1     Running   17         27d   192.168.231.123   k8s-node2    <none>           <none>
    kube-proxy-s7n6w                          1/1     Running   18         27d   192.168.231.121   k8s-master   <none>           <none>
    kube-proxy-xt644                          1/1     Running   0          40s   192.168.231.122   k8s-node1    <none>           <none>
    [root@k8s-master ~]#

k8s-node1上执行 ipvsadm -L -n 还是能看到规则, 在k8s-node1执行如下命令

ip link del kube-ipvs0  
ipvsadm -C

[root@k8s-node1 ~]# ip link
    ...
    11: kube-ipvs0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN mode DEFAULT group default
        link/ether 5e:9b:01:a5:2a:e0 brd ff:ff:ff:ff:ff:ff
    [root@k8s-node1 ~]#
    [root@k8s-node1 ~]# ip link del kube-ipvs0
    [root@k8s-node1 ~]#
    [root@k8s-node1 ~]# ipvsadm -C
    [root@k8s-node1 ~]#
    [root@k8s-node1 ~]# ipvsadm -L -n
    IP Virtual Server version 1.2.1 (size=4096)
    Prot LocalAddress:Port Scheduler Flags
      -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
    [root@k8s-node1 ~]#

使用 kubectl logs kube-proxy-xt644 -n kube-system 查看使用的proxy mode是iptables proxy

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d5X2hoeHg_size_16_color_FFFFFF_t_70 1][]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d5X2hoeHg_size_16_color_FFFFFF_t_70]: /images/20220829/d139c32af7bf45c7aa37ebcf20a41887.png
[20210822222512669.png]: /images/20220829/fc7c6f3c01364ef1821800517f97d62a.png
[20210822225509824.png]: /images/20220829/d9500f9ca5ff40c680c9dad66ab8db6e.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d5X2hoeHg_size_16_color_FFFFFF_t_70 1]: /images/20220829/f6e3a32ef4334ddc88120ec867ed585e.png