Openshift的网络之三：openshift-ovs-subnet代码分析

谁借莪１个温暖的怀抱￠ 2022-05-26 00:59 296阅读 0赞

**一、前言**

Openshift基于OVS SDN的CNI网络方案中，存在一个SDN master节点，而Openshift的所有工作节点也都是SDN的工作节点。

SDN工作节点在启动的时候先向master注册(建立etcd子节点)，master监控到这个节点注册信息之后，会为自节点分配一个子网写入到etcd该子节点下面，然后自节点利用这些信息在节点本地进行初始化配置：

*  建立OVS bridge设备br0，并在br0上设置一些traffic的基本规则，之后所有Openshift启动的POD都可以挂载到这个bridge上面
 *  建立OVS internal port设备tun0，并为这个设备配置SDN子网的网关，然后借助于iptables的NAT，POD就可以通过这个设备访问外部网络。
 *  建立OVS VXLAN设备vxlan\_sys\_4789(vxlan0)，这个设备进行基于vxlan协议的overlay封装，POD借助这个设备访问其它节点的POD。

在SDN工作节点运行期间的职责主要包括：

*  配置POD网络
 *  监听SDN节点的加入和删除事件，更新br0规则以保证访问新的subnet的数据包被转发到vxlan0
 *  监听Openshift project的加入和删除事件，在ovs-multitenant模式下为不同project配置vxlan的VNID，进行流量隔离

在Openshift启动POD的时候，需要SDN工作节点为POD配置网络环境：

*  从子网里面分配一个IP地址给这个POD
 *  将POD的veth对主机侧挂载到br0
 *  修改OVS规则保证不同目的地的数据包可以路由到合适的OVS端口
 *  在ovs-multitenant模式下，添加OVS规则来根据VNID为数据包打包，以满足流量隔离策略

**载自https://blog.csdn.net/cloudvtech**

**二、Openshift节点和网络启动代码分析(Openshift origin v1.5)**

1. /etc/systemd/system/multi-user.target.wants/origin-node.service

2. /usr/bin/openshift start node --config=/etc/origin/node/node-config.yaml --loglevel=5

3.  [https://github.com/openshift/origin/blob/release-1.5/cmd/openshift/openshift.go][https_github.com_openshift_origin_blob_release-1.5_cmd_openshift_openshift.go]

(openshift.CommandFor(basename))

4.  [https://github.com/openshift/origin/blob/release-1.5/pkg/cmd/openshift/openshift.go][https_github.com_openshift_origin_blob_release-1.5_pkg_cmd_openshift_openshift.go]

(cmd = NewCommandOpenShift("openshift”))

5.  [openshift.go][https_github.com_openshift_origin_blob_release-1.5_pkg_cmd_openshift_openshift.go]@ startAllInOne, \_ := start.NewCommandStartAllInOne(name, out, errout)

root.AddCommand(startAllInOne)

6. start\_allinone.go@ startNode, \_ := NewCommandStartNode(basename, out, errout)

7. start\_allinone.go@ startNodeNetwork, \_ := NewCommandStartNetwork(basename, out, errout)

8. start\_node.go@ resolveNodeConfig

9. start\_node.go@ StartNode

Nov 28 06:45:46 ic-node8 origin-node: I1128 06:45:46.677281    1724 start\_node.go:250\] Reading node configuration from /etc/origin/node/node-config.yaml

10. node\_config.go@ BuildKubernetesNodeConfig

let kubelet know the CNI plugin we used

…

KubeletServer: server,

...

![70][]origin/pkg/cmd/server/kubernetes/node.go

start kubelet with config

11. sdnPlugin, err := sdnplugin.NewNodePlugin(options.NetworkConfig.NetworkPluginName, originClient, kubeClient, options.NodeName, options.NodeIP, iptablesSyncPeriod, options.NetworkConfig.MTU)

12. origin/pkg/sdn/plugin/node.go@ NewNodePlugin ( this function returns a instance of NodeConfig)

![70 1][]13. Nov 28 06:45:46 ic-node8 origin-node: I1128 06:45:46.820421    1724 node.go:88\] Initializing SDN node of type "redhat/openshift-ovs-subnet" with configured hostname "ic-node8" (IP ""), iptables sync period “30s"

14 create OVS instance (ovsif, err := ovs.New(kexec.New(), BR, minOvsVersion))

15 calls kubelet initialize function

![70 2][]origin/pkg/sdn/plugin/plugin.go @ func (node \*OsdnNode) Init(host knetwork.Host, hairpinMode componentconfig.HairpinMode, nonMasqueradeCIDR string, mtu int) error \{

origin/vendor/k8s.io/kubernetes/pkg/kubelet/network/plugins.go @ func (plugin \*NoopNetworkPlugin) Init(host Host, hairpinMode componentconfig.HairpinMode, nonMasqueradeCIDR string, mtu int) error \{

Nov 28 06:45:46 ic-node8 origin-node: I1128 06:45:46.933879    1724 plugin.go:30\] openshift-sdn CNI plugin initialized

origin/vendor/k8s.io/kubernetes/pkg/kubelet/network/plugins.go @ func InitNetworkPlugin(plugins \[\]NetworkPlugin, networkPluginName string, host Host, hairpinMode componentconfig.HairpinMode, nonMasqueradeCIDR string, mtu int) (NetworkPlugin, error) \{

Nov 28 06:45:46 ic-node8 origin-node: I1128 06:45:46.934221    1724 plugins.go:181\] Loaded network plugin “cni"

载自https://blog.csdn.net/cloudvtech

三、Openshift SDN 工作接节点运行代码分析

**1. start\_node.go@ StartNode**

if components.Enabled(ComponentPlugins) \{

config.RunPlugin()

**2 origin/pkg/cmd/server/kubernetes/node.go @**

![70 3][]

and calls origin/pkg/sdn/plugin/singletenant.go

**3 origin/pkg/sdn/plugin/node.go @ func (node \*OsdnNode) Start() error \{**

3.1 get network info (ClusterNetwork CIDR and ServiceNetwork CIDR) from Openshift(etcd)

3.2 get local subnet from Openshift(etcd)

3.3 initialize and setup IPTables

3.3.1 newNodeIPTables instance

3.3.2 Setup IPtables (node\_iptables.go @ Setup and syncIPTableRules, iptables will get reload and call a SDN defined relocate func, in this case syncIPTableRules)

3.3.2.1 syncIPTableRules

Nov 28 06:45:46 ic-node8 origin-node: I1128 06:45:46.877751    1724 node\_iptables.go:79\] Syncing openshift iptables rules

3.3.2.1.1 getStaticNodeIPTablesRules

![70 4][]                             3.3.2.1.2 EnsureRule

Nov 28 06:45:46 ic-node8 origin-node: I1128 06:45:46.877778    1724 iptables.go:362\] running iptables -C \[POSTROUTING -t nat -s 10.128.0.0/14 -j MASQUERADE\]

Nov 28 06:45:46 ic-node8 origin-node: I1128 06:45:46.879372    1724 iptables.go:362\] running iptables -I \[POSTROUTING -t nat -s 10.128.0.0/14 -j MASQUERADE\]

Nov 28 06:45:46 ic-node8 origin-node: I1128 06:45:46.880786    1724 iptables.go:362\] running iptables -C \[INPUT -t filter -p udp -m multiport --dports 4789 -m comment --comment 001 vxlan incoming -j ACCEPT\]

Nov 28 06:45:46 ic-node8 origin-node: I1128 06:45:46.933728    1724 iptables.go:362\] running iptables -I \[INPUT -t filter -p udp -m multiport --dports 4789 -m comment --comment 001 vxlan incoming -j ACCEPT\]

Nov 28 06:45:46 ic-node8 origin-node: I1128 06:45:46.939130    1724 iptables.go:362\] running iptables -C \[INPUT -t filter -i tun0 -m comment --comment traffic from SDN -j ACCEPT\]

Nov 28 06:45:46 ic-node8 origin-node: I1128 06:45:46.940629    1724 iptables.go:362\] running iptables -I \[INPUT -t filter -i tun0 -m comment --comment traffic from SDN -j ACCEPT\]

Nov 28 06:45:46 ic-node8 origin-node: I1128 06:45:46.942110    1724 iptables.go:362\] running iptables -C \[INPUT -t filter -i docker0 -m comment --comment traffic from docker -j ACCEPT\]

Nov 28 06:45:46 ic-node8 origin-node: I1128 06:45:46.943486    1724 iptables.go:362\] running iptables -I \[INPUT -t filter -i docker0 -m comment --comment traffic from docker -j ACCEPT\]

Nov 28 06:45:46 ic-node8 origin-node: I1128 06:45:46.944922    1724 iptables.go:362\] running iptables -C \[FORWARD -t filter -d 10.128.0.0/14 -j ACCEPT\]

Nov 28 06:45:46 ic-node8 origin-node: I1128 06:45:46.946310    1724 iptables.go:362\] running iptables -I \[FORWARD -t filter -d 10.128.0.0/14 -j ACCEPT\]

Nov 28 06:45:46 ic-node8 origin-node: I1128 06:45:46.948098    1724 iptables.go:362\] running iptables -C \[FORWARD -t filter -s 10.128.0.0/14 -j ACCEPT\]

Nov 28 06:45:46 ic-node8 origin-node: I1128 06:45:46.955293    1724 iptables.go:362\] running iptables -I \[FORWARD -t filter -s 10.128.0.0/14 -j ACCEPT\]

Nov 28 06:45:46 ic-node8 origin-node: I1128 06:45:46.958209    1724 iptables.go:362\] running iptables -N \[KUBE-MARK-DROP -t nat\]

Nov 28 06:45:46 ic-node8 origin-node: I1128 06:45:46.960938    1724 iptables.go:362\] running iptables -C \[KUBE-MARK-DROP -t nat -j MARK --set-xmark 0x00008000/0x00008000\]

Nov 28 06:45:46 ic-node8 origin-node: I1128 06:45:46.968867    1724 node\_iptables.go:77\] syncIPTableRules took 91.11705ms

3.3.2.2 add IPTables reload hook function (syncIPTableRules)

4 setup SDN (    networkChanged, err := node.SetupSDN())

Nov 28 06:45:46 ic-node8 origin-node: I1128 06:45:46.968901    1724 controller.go:196\] \[SDN setup\] node pod subnet 10.131.2.0/23 gateway 10.131.2.1

Nov 28 06:45:46 ic-node8 origin-node: I1128 06:45:46.970339    1724 controller.go:213\] \[SDN setup\] full SDN setup required

4.1 create bridge: controller.go @ err = plugin.ovs.AddBridge("fail-mode=secure", "protocols=OpenFlow13”) --> origin/pkg/util/ovs/ovs.go @ func (ovsif \*Interface) AddBridge(properties ...string) error \{

Nov 28 06:45:46 ic-node8 origin-node: I1128 06:45:46.970571    1724 ovs.go:62\] Executing: ovs-vsctl --if-exists del-br br0 -- add-br br0 -- set Bridge br0 fail-mode=secure protocols=OpenFlow13

Nov 28 06:45:46 ic-node8 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --if-exists del-br br0 -- add-br br0 -- set Bridge br0 fail-mode=secure protocols=OpenFlow13

4.2 add port vxlan0

\_, err = plugin.ovs.AddPort(VXLAN, 1, "type=vxlan", \`options:remote\_ip="flow"\`, \`options:key="flow”\`)

Nov 28 06:45:47 ic-node8 origin-node: I1128 06:45:47.006615    1724 ovs.go:62\] Executing: ovs-vsctl --may-exist add-port br0 vxlan0 -- set Interface vxlan0 ofport\_request=1 type=vxlan options:remote\_ip="flow" options:key=“flow”

Nov 28 06:45:47 ic-node8 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl --may-exist add-port br0 vxlan0 -- set Interface vxlan0 ofport\_request=1 type=vxlan "options:remote\_ip=\\"flow\\"" "options:key=\\"flow\\""

4.3 add port tun0

\_, err = plugin.ovs.AddPort(TUN, 2, "type=internal”)

Nov 28 06:45:47 ic-node8 origin-node: I1128 06:45:47.045586    1724 ovs.go:62\] Executing: ovs-vsctl --may-exist add-port br0 tun0 -- set Interface tun0 ofport\_request=2 type=internal

4.4 setup open flow table

![70 5][] ![70 6][]    4.5 set up network device

Nov 28 06:45:47 ic-node8 origin-node: I1128 06:45:47.239953    1724 ipcmd.go:44\] Executing: /usr/sbin/ip addr add 10.131.2.1/23 dev tun0

Nov 28 06:45:47 ic-node8 origin-node: I1128 06:45:47.241346    1724 ipcmd.go:44\] Executing: /usr/sbin/ip link set tun0 mtu 1450

Nov 28 06:45:47 ic-node8 origin-node: I1128 06:45:47.242810    1724 ipcmd.go:44\] Executing: /usr/sbin/ip link set tun0 up

Nov 28 06:45:47 ic-node8 NetworkManager\[587\]: <info>  \[1511851547.2442\] device (tun0): link connected

Nov 28 06:45:47 ic-node8 origin-node: I1128 06:45:47.244605    1724 ipcmd.go:44\] Executing: /usr/sbin/ip route add 10.128.0.0/14 dev tun0 proto kernel scope link

Nov 28 06:45:47 ic-node8 origin-node: I1128 06:45:47.256330    1724 ipcmd.go:44\] Executing: /usr/sbin/ip route add 172.30.0.0/16 dev tun0

4.6 add version to flow table

Nov 28 06:45:47 ic-node8 origin-node: I1128 06:45:47.257798    1724 ovs.go:62\] Executing: ovs-ofctl -O OpenFlow13 add-flow br0 table=253, actions=note:00.03

4.7 delete route

defer deleteLocalSubnetRoute(TUN, localSubnetCIDR)

Nov 28 06:45:47 ic-node8 origin-node: I1128 06:45:47.261667    1724 ipcmd.go:44\] Executing: /usr/sbin/ip route show dev tun0

Nov 28 06:45:47 ic-node8 origin-node: I1128 06:45:47.263312    1724 ipcmd.go:44\] Executing: /usr/sbin/ip route del 10.131.2.0/23 dev tun0

5 start subnet change handling task loop

5.1 start the go routing (go utilwait.Forever(node.watchSubnets, 0))

5.2 create an event queue and repopulate the event queue every 30 minutes

origin/pkg/sdn/plugin/subnets.go @ func (node \*OsdnNode) watchSubnets() \{

Nov 28 06:45:47 ic-node8 origin-node: I1128 06:45:47.265598    1724 reflector.go:185\] Starting reflector \*api.HostSubnet (30m0s) from [github.com/openshift/origin/pkg/sdn/plugin/common.go:107][github.com_openshift_origin_pkg_sdn_plugin_common.go_107]

Nov 28 06:45:47 ic-node8 origin-node: I1128 06:45:47.265645    1724 reflector.go:234\] Listing and watching \*api.HostSubnet from [github.com/openshift/origin/pkg/sdn/plugin/common.go:107][github.com_openshift_origin_pkg_sdn_plugin_common.go_107]

the event queue is from Openshift stats cache

![70 7][]

5.3 upon event of k8s host node changes, the process function will be triggered

![70 8][]

5.3.1 on add/sync event

5.3.1.1 add host subnet rules in flow table

Nov 28 06:45:47 ic-node8 origin-node: I1128 06:45:47.268119    1724 subnets.go:294\] Watch Sync event for HostSubnet "ic-node4”

Nov 28 06:45:47 ic-node8 origin-node: I1128 06:45:47.268155    1724 controller.go:427\] AddHostSubnetRules for ic-node4 (host: "ic-node4", ip: "200.222.0.174", subnet: "10.129.2.0/23")

Nov 28 06:45:47 ic-node8 origin-node: I1128 06:45:47.268194    1724 ovs.go:62\] Executing: ovs-ofctl -O OpenFlow13 add-flow br0 table=10, priority=100, tun\_src=200.222.0.174, actions=goto\_table:30

![70 9][]

5.3.2 on delete event

![70 10][]

5.3.2 update multicast rules @ updateVXLANMulticastRules

otx.AddFlow("table=111, priority=100, actions=move:NXM\_NX\_REG0\[\]->NXM\_NX\_TUN\_ID\[0..31\]%s,goto\_table:120", strings.Join(tun\_dsts, ""))

6 start single tenant SDN policy

![70 11][]    case osapi.SingleTenantPluginName:

policy = NewSingleTenantPlugin() // @ origin/pkg/sdn/plugin/singletenant.go

if err = node.policy.Start(node); err != nil \{ // @ origin/pkg/sdn/plugin/node.go

add flow table

otx.AddFlow("table=80, priority=200, actions=output:NXM\_NX\_REG2\[\]")

Nov 28 06:45:47 ic-node8 origin-node: I1128 06:45:47.264897    1724 ovs.go:62\] Executing: ovs-ofctl -O OpenFlow13 add-flow br0 table=80, priority=200, actions=output:NXM\_NX\_REG2\[\]

7 start go routing for K8S service changes  
    7.1 start the go routing (go kwait.Forever(node.watchServices, 0))  
    7.2 create an event queue and repopulate the event queue every 30 minutes  
    7.3 upon event of k8s service changes, the process function will be triggered  
  
        7.3.1 on add/sync event  
Nov 28 06:45:47 ic-node8 origin-node: I1128 06:45:47.271236    1724 node.go:347\] Watch Sync event for Service “prometheus"  
Nov 28 06:45:47 ic-node8 origin-node: I1128 06:45:47.271254    1724 controller.go:457\] AddServiceRules for &TypeMeta\{Kind:,APIVersion:,\}  
Nov 28 06:45:47 ic-node8 origin-node: I1128 06:45:47.271319    1724 ovs.go:62\] Executing: ovs-ofctl -O OpenFlow13 add-flow br0 table=60, tcp, nw\_dst=172.30.85.130, tp\_dst=9090, priority=100, actions=load:0->NXM\_NX\_REG1\[\], load:2->NXM\_NX\_REG2\[\], goto\_table:80  
Nov 28 06:45:47 ic-node8 origin-node: I1128 06:45:47.272503    1724 ovs.go:62\] Executing: ovs-ofctl -O OpenFlow13 add-flow br0 table=50, priority=100, arp, nw\_dst=10.129.2.0/23, actions=move:NXM\_NX\_REG0\[\]->NXM\_NX\_TUN\_ID\[0..31\],set\_field:200.222.0.174->tun\_dst,output:1  
                 7.3.1.1 get vnid according to k8s namespace  
                 7.3.1.2 add rules to flow table  
    return fmt.Sprintf("table=60, %s, nw\_dst=%s, tp\_dst=%d", strings.ToLower(string(protocol)), IP, port)  
    return fmt.Sprintf("%s, priority=100, actions=load:%d->NXM\_NX\_REG1\[\], load:2->NXM\_NX\_REG2\[\], goto\_table:80", baseRule, netID)  
        7.3.2 on delete event  
    return fmt.Sprintf("table=60, %s, nw\_dst=%s, tp\_dst=%d", strings.ToLower(string(protocol)), IP, port)

8 start POD manager  
Nov 28 06:45:47 ic-node8 origin-node: I1128 06:45:47.369006    1724 node.go:244\] Starting openshift-sdn pod manager  
   8.1 get IPAM configure for CNI host-local IPAM plugin use   
(func (m \*podManager) Start(socketPath string, host knetwork.Host, localSubnetCIDR string, clusterNetwork \*net.IPNet) error \{)  
   8.2 start CNI request process go routing (go m.processCNIRequests())  
Nov 28 06:45:47 ic-node8 origin-node: I1128 06:45:47.377831    1724 pod.go:240\] Processing pod network request &\{UPDATE application application-2368900360-nltrr 314aa5ec024dfd329a46ad4d779dfc200a2ba85a288187499c0930988a82795d  0xc421d71740\}  
  
  
![70 12][]  
  
         8.2.1 process the request (CNI ADD/UPDATE/DELETE)  
  
![70 13][]  
                 8.2.1.1 setup/update/teardown is defined at pod\_linux.go  
![70 14][]  
  
                 8.2.1.2 process add request  
        ipamResult, runningPod, err := m.podHandler.setup(request)  
                             8.2.1.2.1 get POD configure (GetVNID, Get(req.PodName), wantsMacvlan(pod), getBandwidth(pod))  
                             8.2.1.2.2 Run CNI IPAM allocation for the container and return the allocated IP address  
                                            result, err := invoke.ExecPluginWithResult("/opt/cni/bin/host-local", m.ipamConfig, args)  
                             8.2.1.2.3 open hostport (m.hostportHandler.OpenPodHostportsAndSync)  
  
载自https://blog.csdn.net/cloudvtech  
                             8.2.1.2.4 setup veth   
                             8.2.1.2.5 configure interface  
Nov 28 06:46:26 ic-node8 NetworkManager\[587\]: <info>  \[1511851586.0352\] manager: (veth2c99ae29): new Veth device (/org/freedesktop/NetworkManager/Devices/11)  
Nov 28 06:46:26 ic-node8 NetworkManager\[587\]: <info>  \[1511851586.0365\] device (veth2c99ae29): link connected  
Nov 28 06:46:26 ic-node8 ovs-vsctl: ovs|00001|vsctl|INFO|Called as ovs-vsctl add-port br0 veth2c99ae29  
Nov 28 06:46:26 ic-node8 NetworkManager\[587\]: <info>  \[1511851586.0530\] device (veth2c99ae29): enslaved to non-master-type device ovs-system; ignoring  
Nov 28 06:46:26 ic-node8 kernel: device veth2c99ae29 entered promiscuous mode  
                             8.2.1.2.6 enable macvlan if needed  
                             8.2.1.2.6 run SDN script  
out, err := exec.Command(sdnScript, setUpCmd, hostVeth.Attrs().Name, contVethMac, podIP.String(), vnidStr, podConfig.ingressBandwidth, podConfig.egressBandwidth).CombinedOutput()  
Nov 28 06:46:26 ic-node8 origin-node: I1128 06:46:26.078507    1724 pod\_linux.go:442\] SetUpPod network plugin output: + lock\_file=/var/lock/openshift-sdn.lock  
 ![70 15][] ![70 16][]  
                             8.2.1.2.7 get ofport  
                             8.2.1.2.8 EnsureVNIDRules  
                 8.2.1.3 process update request  
func (m \*podManager) update(req \*cniserver.PodRequest) (uint32, error) \{  
                             8.2.1.3.1 get namespace  
                             8.2.1.3.2 get POD configure  
                             8.2.1.3.2 get veth info  
                             8.2.1.3.3 exec SDN script  
    out, err := exec.Command(sdnScript, updateCmd, hostVethName, contVethMac, podIP, vnidStr, podConfig.ingressBandwidth, podConfig.egressBandwidth).CombinedOutput()  
                 8.2.1.4 process teardown request  
                             8.2.1.4.1 get veth info  
                             8.2.1.4.2 exec SDN script  
                             8.2.1.4.3 Run CNI IPAM release for the container  
                             8.2.1.4.4 sync hostport changes  
   8.3 start CNI server with handleCNIRequest as handler function  
  
  
container run  time send request to CNI server for POD configure request  
and the request is process as show in 8.2.1  
  
![70 17][]  
  
![70 18][]  
  
  
 9 start vnid sync go routing (go kwait.Forever(node.policy.SyncVNIDRules, time.Hour))  
  
  
10 openshift-sdn ready  
Nov 28 06:45:47 ic-node8 origin-node: I1128 06:45:47.382565    1724 node.go:267\] openshift-sdn network plugin ready

载自https://blog.csdn.net/cloudvtech

[https_github.com_openshift_origin_blob_release-1.5_cmd_openshift_openshift.go]: https://github.com/openshift/origin/blob/release-1.5/cmd/openshift/openshift.go
[https_github.com_openshift_origin_blob_release-1.5_pkg_cmd_openshift_openshift.go]: https://github.com/openshift/origin/blob/release-1.5/pkg/cmd/openshift/openshift.go
[70]: /images/20220526/c057ef3baac24215b03a93298cc527e0.png
[70 1]: /images/20220526/6bde6666343d4a058d53580f4305b846.png
[70 2]: /images/20220526/e95695306d3b4f7395a9ce1ad8618237.png
[70 3]: /images/20220526/fcecf4c41a0a42c2a1cadfdc3c15d907.png
[70 4]: /images/20220526/abe60ca2b6a94c9ab603015d42a61190.png
[70 5]: https://img-blog.csdn.net/20180423140053166?watermark/2/text/aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2Nsb3VkdnRlY2g=/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70
[70 6]: /images/20220526/206a28cca95e45daa2f1e8184efa968a.png
[github.com_openshift_origin_pkg_sdn_plugin_common.go_107]: http://github.com/openshift/origin/pkg/sdn/plugin/common.go:107
[70 7]: /images/20220526/8bcd8b44a57a4c0b91950cef5b064329.png
[70 8]: /images/20220526/0ab4031109b54652a0fc08909ffdaf98.png
[70 9]: /images/20220526/c40baa5276d54db9b26118f52563948c.png
[70 10]: /images/20220526/1a36eb80b4d843f99ea48565b85dec85.png
[70 11]: /images/20220526/3f64a7d6071a471ca7bb4c32c6c16488.png
[70 12]: /images/20220526/5cb699f80d294102b3402d94fbbe0f11.png
[70 13]: /images/20220526/25dd2604c7ca42eda1f9e618c233be0b.png
[70 14]: /images/20220526/21a5299cccee4db7b02e72dd113207d4.png
[70 15]: https://img-blog.csdn.net/20180423135415649?watermark/2/text/aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2Nsb3VkdnRlY2g=/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70
[70 16]: /images/20220526/e0845e0036484f93abfeb5e5aab3eacb.png
[70 17]: /images/20220526/1be7778d9e1f4468bdb5323862646c2c.png
[70 18]: /images/20220526/642b4b5375c64676beab2040af72eba9.png