DPDK系列之十四：在kubernetes运行Intel的vhost-user-net-plugin插件以支持dpdk

r囧r小猫 2022-05-23 00:48 222阅读 0赞

**一、前言**

根据文章[《容器云的数据通道加速方案概览》][Link 1]的描述，借助Intel提供的[vhost-user-net-plugin][] CNI，可以为基于Kubernetes的容器配置使用virtio前端和vhost-user后端的数据通道加速方案，vhost-user后端port可以由OVS-DPDK或者VPP提供。

本文就将使用multus绑定flannel和vhost-user-net-plugin两种网络类型，为容器提供除了传统网络以外DPDK的支持。

multus的配置可以参照文章[《kubernetes系列之十三：POD多网卡方案multus-cni之通过CRD给POD配置自由组合网络》][kubernetes_POD_multus-cni_CRD_POD]。

转载自https://blog.csdn.net/cloudvtech

**二、编译vhost-user-net-plugin CNI**

2.1 参照git文档

github地址：https://github.com/intel/vhost-user-net-plugin

https://github.com/intel/vhost-user-net-plugin.git

git clone https://github.com/intel/vhost-user-net-plugin.git
    cd vhost-user-net-plugin
    ./build

build出来的binary就在bin目录

[root@k8s-master vhost-user-net-plugin]# ls -l bin
    总用量 3108
    -rwxr-xr-x 1 root root 3179579 5月  29 07:56 vhostuser

2.2 安装vhost-user-net-plugin CNI

将vhost-user-net-plugin目录下bin/vhostuser文件和tests目录拷贝到node的/opt/cni/bin目录

2.3 node上的CNI配置文件

本测试只使用一个master和一个node

rm -f /etc/cni/net.d/*

创建新的multus CRD conf如下：

/etc/cni/net.d/10-multus.conf 
    {  
      "name": "multus-cni-network",  
      "type": "multus",  
      "kubeconfig": "/etc/kubernetes/kubelet.conf"  
    }

重启kubelet

systemctl restart kubelet

转载自https://blog.csdn.net/cloudvtech

**三、编译并运行OVS-DPDK**

参见文章[《DPD系列之一：DPDK 17.08.1在CentoS 7.2.1511的安装》][DPD_DPDK 17.08.1_CentoS 7.2.1511]和文章[《DPDK系列之七：OVS-DPDK的在CentOS安装和测试》][DPDK_OVS-DPDK_CentOS]。

转载自https://blog.csdn.net/cloudvtech

**四、编译容器镜像**

参见文章[《DPDK系列之十三：容器基于OVS-DPDK的dpdkvhostuser端口的pktgen和testpmd数据包测试》][DPDK_OVS-DPDK_dpdkvhostuser_pktgen_testpmd]。

这里只要使用testpmd容器就可以了。

转载自https://blog.csdn.net/cloudvtech

**五、部署vhost-user-net-plugin网络**

5.1 multus Network CRD yml文件

ovs-dpdk-network.yaml 
    apiVersion: "kubernetes.com/v1"  
    kind: Network  
    metadata:  
      name: ovs-dpdk-networkobj  
    plugin: vhostuser  
    args: '[  
    {
      "type": "vhostuser",
      "name": "vhostuser-network",
      "if0name": "net0",
      "vhost": {
        "vhost_tool": "/opt/cni/bin/tests/ovs-config.py"
      },
      "ipam": {
        "type": "host-local",
        "subnet": "192.168.166.0/24",
        "rangeStart": "192.168.166.220",
        "rangeEnd": "192.168.166.240",
        "routes": [
          {
            "dst": "0.0.0.0/0"
          }
        ],
        "gateway": "192.168.166.2"
      }
    }
    ]'

使用vhost-user-net-plugin tests目录下ovs-config.py，这个tool的主要功能就是

使用vs-vsctl为POD创建一个dpdkvhostuser port并且将socket移动到/var/lib/cni/vhostuser/目录

![70][]

5.2 部署vhost-user-net CNI网络

[root@k8s-master CDR]# kubectl get crd
    NAME                      AGE
    networks.kubernetes.com   21d
    [root@k8s-master CDR]# kubectl get networks
    NAME                  AGE
    flannel-networkobj    21d
    macvlan-networkobj    21d
    ovs-dpdk-networkobj   3h
    [root@k8s-master CDR]# kubectl describe networks ovs-dpdk-networkobj
    Name:         ovs-dpdk-networkobj
    Namespace:    default
    Labels:       <none>
    Annotations:  kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"kubernetes.com/v1","args":"[ { \"type\": \"vhostuser\", \"name\": \"vhostuser-network\", \"if0name\": \"net0\", \"vhost\": { \"vhost_too...
    API Version:  kubernetes.com/v1
    Args:         [ { "type": "vhostuser", "name": "vhostuser-network", "if0name": "net0", "vhost": { "vhost_tool": "/opt/cni/bin/tests/ovs-config.py" }, "ipam": { "type": "host-local", "subnet": "192.168.166.0/24", "rangeStart": "192.168.166.220", "rangeEnd": "192.168.166.240", "routes": [ { "dst": "0.0.0.0/0" } ], "gateway": "192.168.166.2" } } ]
    Kind:         Network
    Metadata:
      Cluster Name:        
      Creation Timestamp:  2018-05-29T12:20:31Z
      Generation:          1
      Resource Version:    356234
      Self Link:           /apis/kubernetes.com/v1/namespaces/default/networks/ovs-dpdk-networkobj
      UID:                 adfbbf3a-633a-11e8-8aca-000c29d3e746
    Plugin:                vhostuser
    Events:                <none>
    [root@k8s-master CDR]#

5.3 POD配置

cat testpmd_pod_flannel_vhostuser.yml 
    apiVersion: v1
    kind: Pod
    metadata:
      name: nginx-flannel-vhostuser
      annotations:
        networks: '[  
            { "name": "flannel-networkobj" },
            { "name": "ovs-dpdk-networkobj" }
        ]'
    spec:
      containers:
          - name: ovs-dpdk
            image: dpdk_testpmd_1708
            imagePullPolicy: Never
            securityContext:
              privileged: true
            command: [ "/bin/bash", "-c", "--" ]
            args: [ "while true; do sleep 30; done;" ]
            volumeMounts:
             - name: hugepage
               mountPath: /mnt/huge
             - name: socketdir
               mountPath: /var/run/openvswitch
      volumes:
      - name: hugepage
        hostPath:
         path: /mnt/huge
      - name: socketdir
        hostPath:
         path: /var/lib/cni/vhostuser

部署两个POD

[root@k8s-master CDR]# kubectl apply -f testpmd_pod_flannel_vhostuser.yml 
    pod "nginx-flannel-vhostuser" created
    [root@k8s-master CDR]# kubectl apply -f testpmd_pod_flannel_vhostuser1.yml 
    pod "nginx-flannel-vhostuser1" created

转载自https://blog.csdn.net/cloudvtech

**六、查看部署结果**

6.1 log可以看到建立了两个port

6.2 POD状态

[root@k8s-master CDR]# kubectl get pod -o wide
    NAME                                READY     STATUS    RESTARTS   AGE       IP            NODE
    nginx-flannel-vhostuser             1/1       Running   0          10m       10.244.1.14   k8s-node1
    nginx-flannel-vhostuser1            1/1       Running   0          10m       10.244.1.13   k8s-node1

[root@k8s-master CDR]# ping 10.244.1.14
    PING 10.244.1.14 (10.244.1.14) 56(84) bytes of data.
    64 bytes from 10.244.1.14: icmp_seq=1 ttl=63 time=0.504 ms
    64 bytes from 10.244.1.14: icmp_seq=2 ttl=63 time=0.466 ms
    64 bytes from 10.244.1.14: icmp_seq=3 ttl=63 time=0.512 ms
    ^C
    --- 10.244.1.14 ping statistics ---
    3 packets transmitted, 3 received, 0% packet loss, time 2000ms
    rtt min/avg/max/mdev = 0.466/0.494/0.512/0.020 ms

6.3 node上容器状态

[root@k8s-node1 ~]# docker ps | grep bash
    4663c3f744ff        284b78f4e48f                 "/bin/bash -c -- '..."   3 minutes ago       Up 3 minutes                            k8s_ovs-dpdk_nginx-flannel-vhostuser_default_31243b57-635a-11e8-8aca-000c29d3e746_0
    668df5db09c6        284b78f4e48f                 "/bin/bash -c -- '..."   3 minutes ago       Up 3 minutes                            k8s_ovs-dpdk_nginx-flannel-vhostuser1_default_2fefba60-635a-11e8-8aca-000c29d3e746_0

[root@k8s-node1 ~]# docker exec -it 4663c3f744ff bash
    [root@nginx-flannel-vhostuser dpdk-17.05]# ifconfig 
    eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
            inet 10.244.1.14  netmask 255.255.255.0  broadcast 0.0.0.0
            inet6 fe80::c25:caff:fe9d:5b8d  prefixlen 64  scopeid 0x20<link>
            ether 0a:58:0a:f4:01:0e  txqueuelen 0  (Ethernet)
            RX packets 13  bytes 962 (962.0 B)
            RX errors 0  dropped 0  overruns 0  frame 0
            TX packets 8  bytes 648 (648.0 B)
            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
    
    lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
            inet 127.0.0.1  netmask 255.0.0.0
            inet6 ::1  prefixlen 128  scopeid 0x10<host>
            loop  txqueuelen 1  (Local Loopback)
            RX packets 0  bytes 0 (0.0 B)
            RX errors 0  dropped 0  overruns 0  frame 0
            TX packets 0  bytes 0 (0.0 B)
            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

6.4 vhost socket

POD的网络容器pause状态如下：

d7157cc4f590        k8s.gcr.io/pause-amd64:3.1   "/pause"                 3 minutes ago       Up 3 minutes                            k8s_POD_nginx-flannel-vhostuser_default_31243b57-635a-11e8-8aca-000c29d3e746_0
    70dc0863789f        k8s.gcr.io/pause-amd64:3.1   "/pause"                 3 minutes ago       Up 3 minutes                            k8s_POD_nginx-flannel-vhostuser1_default_2fefba60-635a-11e8-8aca-000c29d3e746_0

所以POD的vhost socket应该在/var/lib/cni/vhostuser目录，对应两个pause容器ID

[root@k8s-node1 ~]# ls -l /var/lib/cni/vhostuser/
    总用量 0
    drwx------ 2 root root 95 5月  29 12:12 70dc0863789fd13a61ad14fc333a50175992cc31727e204c14c6fa53f241885a
    drwx------ 2 root root 95 5月  29 12:12 d7157cc4f590a84c0e460daf8062b44a1c4b80d4ec6213551915dd35527caf68

[root@k8s-node1 70dc0863789fd13a61ad14fc333a50175992cc31727e204c14c6fa53f241885a]# ls -l
    总用量 8
    srwxr-xr-x 1 root root   0 5月  29 12:12 70dc0863789f-net0
    -rw-r--r-- 1 root root 127 5月  29 12:12 70dc0863789f-net0-ip4.conf
    -rw-r--r-- 1 root root 156 5月  29 12:12 70dc0863789f-net0.json

[root@k8s-node1 70dc0863789fd13a61ad14fc333a50175992cc31727e204c14c6fa53f241885a]# cat 70dc0863789f-net0-ip4.conf 
    {
    	"ipAddr": "192.168.166.228/32",
    	"macAddr": "8e:10:52:bf:e5:70",
    	"gateway": "169.254.1.1",
    	"gwMac": "00:00:00:00:00:00"
    }
    [root@k8s-node1 70dc0863789fd13a61ad14fc333a50175992cc31727e204c14c6fa53f241885a]# cat 70dc0863789f-net0.json 
    {"vhostname":"70dc0863789f-net0","vhostmac":"00:00:00:00:00:00","ifname":"net0","ifmac":"8e:10:52:bf:e5:70","vhost_tool":"/opt/cni/bin/tests/ovs-config.py"}

6.5 OVS-DPDK port

/root/openvswitch-2.8.1/utilities/ovs-vsctl show
    590c0cb0-706c-4d36-a9ed-611a824a5a9e
        Bridge "br0"
            Port "70dc0863789f-net0"
                Interface "70dc0863789f-net0"
                    type: dpdkvhostuser
            Port "d7157cc4f590-net0"
                Interface "d7157cc4f590-net0"
                    type: dpdkvhostuser
            Port "br0"
                Interface "br0"
                    type: internal

转载自https://blog.csdn.net/cloudvtech

[Link 1]: https://blog.csdn.net/cloudvtech/article/details/80408099
[vhost-user-net-plugin]: https://github.com/intel/vhost-user-net-plugin
[kubernetes_POD_multus-cni_CRD_POD]: https://blog.csdn.net/cloudvtech/article/details/80238843
[DPD_DPDK 17.08.1_CentoS 7.2.1511]: https://blog.csdn.net/cloudvtech/article/details/80221236
[DPDK_OVS-DPDK_CentOS]: https://blog.csdn.net/cloudvtech/article/details/80393032
[DPDK_OVS-DPDK_dpdkvhostuser_pktgen_testpmd]: https://blog.csdn.net/cloudvtech/article/details/80492234
[70]: /images/20220523/9ac88ee80ffd47fa8d5ffc735a3c230d.png