dive into openstack ovn (by quqi99)

旧城等待， 2022-11-05 08:37 214阅读 0赞

**作者：张华 发表于：2021-03-04  
版权声明：可以任意转载，转载时请务必以超链接形式标明文章原始出处和作者信息及本版权声明**

前一篇基础是：Play with OVN - https://blog.csdn.net/quqi99/article/details/103194137  
这一篇将主要讲openstack如何来使用ovn的。  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3F1cWk5OQ_size_16_color_FFFFFF_t_70_pic_center]

## 测试环境 ##

下面将搭建一下类似的测试环境。

./generate-bundle.sh --name ovn --series bionic --release ussuri --ovn --vault --create-model --run
    juju add-unit nova-compute
    ./configure
    source novarc
    
    neutron net-create private2 --provider:network_type geneve --provider:segmentation_id 1012
    neutron subnet-create --gateway 192.168.22.1 private2 192.168.22.0/24 --enable_dhcp=True --name private_subnet2
    ROUTER_ID=$(neutron router-list |grep ' provider-router ' |awk '{print $2}')
    SUBNET_ID=$(neutron subnet-list |grep '192.168.22.0/24' |awk '{print $2}')
    neutron router-interface-add $ROUTER_ID $SUBNET_ID
    nova hypervisor-list
    openstack server create --wait --image bionic --flavor m1.small --key-name testkey --nic net-id=$(openstack net show private -f value -c id) --availability-zone=nova:juju-c40d4b-ovn-13.cloud.sts i1
    ./tools/float_all.sh
    ./tools/sec_groups.sh
    openstack server create --wait --image cirros2 --flavor m1.small --key-name testkey --nic net-id=$(openstack net show private -f value -c id) --availability-zone=nova:juju-c40d4b-ovn-6.cloud.sts i2
    openstack server create --wait --image cirros2 --flavor m1.small --key-name testkey --nic net-id=$(openstack net show private2 -f value -c id) --availability-zone=nova:juju-c40d4b-ovn-6.cloud.sts i3
    
    $ nova list
    +--------------------------------------+------+--------+------------+-------------+--------------------------------------+
    | ID                                   | Name | Status | Task State | Power State | Networks                             |
    +--------------------------------------+------+--------+------------+-------------+--------------------------------------+
    | 82c89129-0335-4e33-b117-be940a7020d4 | i1   | ACTIVE | -          | Running     | private=192.168.21.161, 10.5.150.115 |
    | 74641e74-3401-44f9-8d7d-bef3ea0fdb92 | i2   | ACTIVE | -          | Running     | private=192.168.21.3                 |
    | 37f7c5c0-844c-4d8d-ad95-aa29b7418dc0 | i3   | ACTIVE | -          | Running     | private2=192.168.22.47               |
    +--------------------------------------+------+--------+------------+-------------+--------------------------------------+

## OVN Northbound DB与Neutron的概念映射 ##

Neutron中有Network, Subnet, Router, Port的概念，OVN Northbound DB中也有对应的逻辑概念: Switch=Neutron Subnet, Port=Neutron Port, Distributed Router=Neutron DVR Router, Gateway Router=Neutron Centralized L3, Port=Neutron Port

## 举例查看OVN Northbound DB中和L3 NAT相关的数据 ##

#run in compute node
    # ovs-vsctl get open . external_ids
    {hostname=juju-c40d4b-ovn-6.cloud.sts, ovn-bridge-mappings="physnet1:br-data", ovn-cms-options=enable-chassis-as-gw, ovn-encap-ip="10.5.0.178", ovn-encap-type=geneve, ovn-remote="ssl:10.5.2.178:6642,ssl:10.5.1.220:6642,ssl:10.5.1.157:6642", rundir="/var/run/openvswitch", system-id=juju-c40d4b-ovn-6.cloud.sts}
    export SB=$(sudo ovs-vsctl get open . external_ids:ovn-remote | sed -e 's/\"//g')
    export NB=$(sudo ovs-vsctl get open . external_ids:ovn-remote | sed -e 's/\"//g' | sed -e 's/6642/6641/g')
    
    #从所有计算节点上的ovn-controller中所出用于中心化l3的那个(搜索enable-chassis-as-gw得知是juju-c40d4b-ovn-6.cloud.sts(uuid=f8004279-14d2-48fd-8b6a-f025706fa8a8)
    #run in ovnnb_db master and ovnsb_db master
    juju ssh ovn-central/1 -- sudo -s
    root@juju-c40d4b-ovn-8:~# ovn-sbctl list chassis
    _uuid               : add1028c-e19c-4f23-8795-a0c64f16fdcd
    encaps              : [7417dd02-d2d5-45bb-88b9-3dafe07c92c6]
    external_ids        : {datapath-type=system, iface-types="erspan,geneve,gre,internal,ip6erspan,ip6gre,lisp,patch,stt,system,tap,vxlan", is-interconn="false", neutron-metadata-proxy-networks="d0382b73-eb07-4314-a803-b957662f618c", "neutron:liveness_check_at"="2021-03-04T10:34:37.869388+00:00", "neutron:metadata_liveness_check_at"="2021-03-04T10:34:38.163481+00:00", "neutron:ovn-metadata-id"="4849b5a6-3134-4ca2-9fea-38c24aef6121", "neutron:ovn-metadata-sb-cfg"="578", ovn-bridge-mappings="", ovn-chassis-mac-mappings="", ovn-cms-options=""}
    hostname            : juju-c40d4b-ovn-13.cloud.sts
    name                : juju-c40d4b-ovn-13.cloud.sts
    nb_cfg              : 578
    transport_zones     : []
    vtep_logical_switches: []
    _uuid               : f8004279-14d2-48fd-8b6a-f025706fa8a8
    encaps              : [50fde256-2d20-4e4d-aa6e-c7838edee407]
    external_ids        : {datapath-type=system, iface-types="erspan,geneve,gre,internal,ip6erspan,ip6gre,lisp,patch,stt,system,tap,vxlan", is-interconn="false", neutron-metadata-proxy-networks="d0382b73-eb07-4314-a803-b957662f618c,f1b85533-3f78-4f44-9785-996e725bb3bf", "neutron:liveness_check_at"="2021-03-04T10:34:37.322922+00:00", "neutron:metadata_liveness_check_at"="2021-03-04T10:34:37.602977+00:00", "neutron:ovn-metadata-id"="9dd62f6d-c41b-42e2-b424-7d0bbf0902ea", "neutron:ovn-metadata-sb-cfg"="578", ovn-bridge-mappings="physnet1:br-data", ovn-chassis-mac-mappings="", ovn-cms-options=enable-chassis-as-gw}
    hostname            : juju-c40d4b-ovn-6.cloud.sts
    name                : juju-c40d4b-ovn-6.cloud.sts
    nb_cfg              : 578
    transport_zones     : []
    vtep_logical_switches: []
    
    #或者直接找到它l3 ovn-controller
    root@juju-c40d4b-ovn-8:~# ovn-nbctl list Gateway_Chassis
    _uuid               : 5de561d4-77e5-467d-8b69-ec064e949d8c
    chassis_name        : juju-c40d4b-ovn-6.cloud.sts
    external_ids        : {}
    name                : lrp-304cfe5a-d25c-41aa-bfbe-9ba60c7248c2_juju-c40d4b-ovn-6.cloud.sts
    options             : {}
    priority            : 1
    
    
    #找到neutron router的external_gateway_info和routerid=1a4bf8d2-c885-4af8-8b9c-061c7b27fa69
    $ openstack router show provider-router --fit-width |grep external_gateway_info
    | external_gateway_info   | {"network_id": "1d7749fd-90c9-4f31-ada4-50f1845ca32e", "external_fixed_ips": [{"subnet_id": "4009f18b-eb09-4b74-a0ac-ce29537838a3", "ip_address": "10.5.152.46"}], "enable_snat": true}       
    
    #找到与此neutron router对应的ovn router
    root@juju-c40d4b-ovn-8:~# ovn-nbctl find Logical_Router name=neutron-1a4bf8d2-c885-4af8-8b9c-061c7b27fa69
    _uuid               : 89a12d3b-28ad-466b-97d6-971c669aee44
    enabled             : true
    external_ids        : {"neutron:availability_zone_hints"="", "neutron:gw_port_id"="304cfe5a-d25c-41aa-bfbe-9ba60c7248c2", "neutron:revision_number"="5", "neutron:router_name"=provider-router}
    load_balancer       : []
    name                : neutron-1a4bf8d2-c885-4af8-8b9c-061c7b27fa69
    nat                 : [6196ba5f-568b-486e-b7d6-add825d2f8f9, b1e5878e-95f0-45f7-b3a2-232b550be281, cb82abf5-55b5-4731-aa24-b993ac4621d9]
    options             : {}
    policies            : []
    ports               : [3aa3c7ce-ec0f-4d3e-9a3b-15ae7f750622, f964314d-ac73-4dda-a940-9c6910850b34, fb90ec4c-ac5e-4415-96e8-28ea18e53205]
    static_routes       : [cd814869-e8a0-4dde-9730-362d5d83a1d0]
    
    #从OVN NAT northbound表中验证SNAT
    root@juju-c40d4b-ovn-8:~# ovn-nbctl find NAT type=snat
    _uuid               : b1e5878e-95f0-45f7-b3a2-232b550be281
    external_ids        : {}
    external_ip         : "10.5.152.46"
    external_mac        : []
    logical_ip          : "192.168.22.0/24"
    logical_port        : []
    options             : {}
    type                : snat
    _uuid               : 6196ba5f-568b-486e-b7d6-add825d2f8f9
    external_ids        : {}
    external_ip         : "10.5.152.46"
    external_mac        : []
    logical_ip          : "192.168.21.0/24"
    logical_port        : []
    options             : {}
    type                : snat
    
    root@juju-c40d4b-ovn-8:~# ovn-nbctl find NAT type=dnat_and_snat
    _uuid               : cb82abf5-55b5-4731-aa24-b993ac4621d9
    external_ids        : {"neutron:fip_external_mac"="fa:16:3e:b6:11:c5", "neutron:fip_id"="9bf3a29c-e7fe-4dca-8c59-a8809ae87db9", "neutron:fip_port_id"="13d0a59c-e25d-48f5-af68-ca18dbbf139d", "neutron:revision_number"="2", "neutron:router_name"=neutron-1a4bf8d2-c885-4af8-8b9c-061c7b27fa69}
    external_ip         : "10.5.150.115"
    external_mac        : []
    logical_ip          : "192.168.21.161"
    logical_port        : "13d0a59c-e25d-48f5-af68-ca18dbbf139d"
    options             : {}
    type                : dnat_and_snat
    
    root@juju-c40d4b-ovn-8:~# ovn-nbctl lr-nat-list neutron-1a4bf8d2-c885-4af8-8b9c-061c7b27fa69
    TYPE             EXTERNAL_IP        LOGICAL_IP            EXTERNAL_MAC         LOGICAL_PORT
    dnat_and_snat    10.5.150.115       192.168.21.161
    snat             10.5.152.46        192.168.21.0/24
    snat             10.5.152.46        192.168.22.0/24

## OVN Southbound DB - L2 Logical Flow (同网段大二层) ##

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3F1cWk5OQ_size_16_color_FFFFFF_t_70_pic_center 1]

对于vm1=i1(192.168.21.161)访问vm3=i3(192.168.21.3)

1, port security只是对从虚机进来的包检查它的IP与MAC是否对应

root@juju-c40d4b-ovn-8:~# ovn-sbctl lflow-list |grep inport |grep 13d0a59c-e25d-48f5-af68-ca18dbbf139d |grep -E 'table=0'
      table=0 (ls_in_port_sec_l2  ), priority=50   , match=(inport == "13d0a59c-e25d-48f5-af68-ca18dbbf139d" && eth.src == {fa:16:3e:54:36:ad}), action=(next;)

接下来的看这篇文章 -  
OpenStack SDN With OVN (Part 2) - Network Engineering Analysis  
https://networkop.co.uk/blog/2016/12/10/ovn-part2/

## OVN Southbound DB - L3 Logical Flow(南北) ##

![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3F1cWk5OQ_size_16_color_FFFFFF_t_70_pic_center 2]  
见　－　 https://networkop.co.uk/blog/2016/12/10/ovn-part2/

## OVN Southbound DB - ovn controller Logical Flow (不同网段东西) ##

## 通过ovn-trace调试OVN Sourthbound DB逻辑流 ##

ovn-trace能用来帮助调试或者理解上面的ovn南向逻辑流

root@juju-c40d4b-ovn-8:~# ovn-sbctl lflow-list |grep -i datapath
    Datapath: "neutron-1a4bf8d2-c885-4af8-8b9c-061c7b27fa69" aka "provider-router" (587cfb5a-2797-405f-83f0-385211e2ad78)  Pipeline: ingress
    Datapath: "neutron-1a4bf8d2-c885-4af8-8b9c-061c7b27fa69" aka "provider-router" (587cfb5a-2797-405f-83f0-385211e2ad78)  Pipeline: egress
    Datapath: "neutron-1d7749fd-90c9-4f31-ada4-50f1845ca32e" aka "ext_net" (74359f20-4009-4c7e-afd8-3f3dd2423b77)  Pipeline: ingress
    Datapath: "neutron-1d7749fd-90c9-4f31-ada4-50f1845ca32e" aka "ext_net" (74359f20-4009-4c7e-afd8-3f3dd2423b77)  Pipeline: egress
    Datapath: "neutron-b2f023b1-4a05-4443-b334-cf47e90a1567" aka "private" (d0382b73-eb07-4314-a803-b957662f618c)  Pipeline: ingress
    Datapath: "neutron-b2f023b1-4a05-4443-b334-cf47e90a1567" aka "private" (d0382b73-eb07-4314-a803-b957662f618c)  Pipeline: egress
    Datapath: "neutron-1c537bdd-5633-4263-a364-b14cecd4e92d" aka "private2" (f1b85533-3f78-4f44-9785-996e725bb3bf)  Pipeline: ingress
    Datapath: "neutron-1c537bdd-5633-4263-a364-b14cecd4e92d" aka "private2" (f1b85533-3f78-4f44-9785-996e725bb3bf)  Pipeline: egress

root@juju-c40d4b-ovn-8:~# ovn-nbctl show c6cc6d66-91af-4613-87aa-cbd770d8040d
    switch c6cc6d66-91af-4613-87aa-cbd770d8040d (neutron-b2f023b1-4a05-4443-b334-cf47e90a1567) (aka private)
        port abe38147-7909-4708-ad02-d478e62e7ff1
            type: router
            router-port: lrp-abe38147-7909-4708-ad02-d478e62e7ff1
        port 13d0a59c-e25d-48f5-af68-ca18dbbf139d
            addresses: ["fa:16:3e:54:36:ad 192.168.21.161"]
        port 0a2c4125-791c-4824-837a-1a940e78673a
            type: localport
            addresses: ["fa:16:3e:2d:ce:27 192.168.21.2"]
        port cd9fefdb-00f0-4efd-950b-84ba32788571
            addresses: ["fa:16:3e:78:2f:34 192.168.21.3"]

例如上面的192.168.21.161如何访问192.168.21.3的.

root@juju-c40d4b-ovn-8:~# ovn-trace --minimal neutron-b2f023b1-4a05-4443-b334-cf47e90a1567 'inport == "13d0a59c-e25d-48f5-af68-ca18dbbf139d" && eth.src == fa:16:3e:54:36:ad && eth.dst == fa:16:3e:78:2f:34'
    # reg14=0x3,vlan_tci=0x0000,dl_src=fa:16:3e:54:36:ad,dl_dst=fa:16:3e:78:2f:34,dl_type=0x0000
    output("cd9fef");
    
    root@juju-c40d4b-ovn-8:~# ovn-trace  neutron-b2f023b1-4a05-4443-b334-cf47e90a1567 'inport == "13d0a59c-e25d-48f5-af68-ca18dbbf139d" && eth.src == fa:16:3e:54:36:ad && eth.dst == fa:16:3e:78:2f:34'
    # reg14=0x3,vlan_tci=0x0000,dl_src=fa:16:3e:54:36:ad,dl_dst=fa:16:3e:78:2f:34,dl_type=0x0000
    
    ingress(dp="private", inport="13d0a5")
    --------------------------------------
     0. ls_in_port_sec_l2 (ovn-northd.c:4516): inport == "13d0a5" && eth.src == {fa:16:3e:54:36:ad}, priority 50, uuid 620c23f4
        next;
    19. ls_in_l2_lkup (ovn-northd.c:6779): eth.dst == fa:16:3e:78:2f:34, priority 50, uuid cdf396b9
        outport = "cd9fef";
        output;
    
    egress(dp="private", inport="13d0a5", outport="cd9fef")
    -------------------------------------------------------
     9. ls_out_port_sec_l2 (ovn-northd.c:4582): outport == "cd9fef" && eth.dst == {fa:16:3e:78:2f:34}, priority 50, uuid b48eb374
        output;
        /* output to "cd9fef", type "" */
    root@juju-c40d4b-ovn-8:~# ovn-trace --summary neutron-b2f023b1-4a05-4443-b334-cf47e90a1567 'inport == "13d0a59c-e25d-48f5-af68-ca18dbbf139d" && eth.src == fa:16:3e:54:36:ad && eth.dst == fa:16:3e:78:2f:34'
    # reg14=0x3,vlan_tci=0x0000,dl_src=fa:16:3e:54:36:ad,dl_dst=fa:16:3e:78:2f:34,dl_type=0x0000
    ingress(dp="private", inport="13d0a5") {
        next;
        outport = "cd9fef";
        output;
        egress(dp="private", inport="13d0a5", outport="cd9fef") {
            output;
            /* output to "cd9fef", type "" */;
        };
    };

再例如如何访问8.8.8.8

root@juju-c40d4b-ovn-8:~# ovn-trace --detail neutron-b2f023b1-4a05-4443-b334-cf47e90a1567 'inport == "13d0a59c-e25d-48f5-af68-ca18dbbf139d" && eth.src == fa:16:3e:54:36:ad && ip4.dst == 8.8.8.8'
    # ip,reg14=0x3,vlan_tci=0x0000,dl_src=fa:16:3e:54:36:ad,dl_dst=00:00:00:00:00:00,nw_src=0.0.0.0,nw_dst=8.8.8.8,nw_proto=0,nw_tos=0,nw_ecn=0,nw_ttl=0
    
    ingress(dp="private", inport="13d0a5")
    --------------------------------------
     0. ls_in_port_sec_l2 (ovn-northd.c:4516): inport == "13d0a5" && eth.src == {fa:16:3e:54:36:ad}, priority 50, uuid 620c23f4
        next;
     1. ls_in_port_sec_ip (ovn-northd.c:4225): inport == "13d0a5" && eth.src == fa:16:3e:54:36:ad && ip, priority 80, uuid 47d5e0e8
        drop;

## 计算节点上的openflow flow ##

上面的都是OVN南向DB的逻辑流，直接到计算节点上的openflow通过“ovs-ofctl dump-flows br-int”查看，可通过“ovs-appctl ofproto/trace”来调试(ovn Logical Flow流过ovn-trace调试)，见：https://blog.russellbryant.net/2016/11/11/ovn-logical-flows-and-ovn-trace/

如：  
sudo ovs-appctl ofproto/trace br-int in\_port=6,arp,arp\_spa=192.168.21.7,dl\_src=fa:16:3e:c4:58:9c  
可使用ovs-stat snap工具来方便生成ovs-appctl辅助命令，见：　https://blog.csdn.net/quqi99/article/details/111831695

## 实操 - 虚机访问外部UDP服务调试流表 ##

找到了centralized l3为juju-c40d4b-ovn-6.cloud.sts，　
    root@juju-c40d4b-ovn-6:~# sudo ovs-appctl dpif/show
    system@ovs-system: hit:243081 missed:5389
      br-data:
        br-data 65534/2: (internal)
        ens8 1/3: (system)
        patch-provnet-5a77e708-5c1f-48dc-acdf-21e66d8e3be7-to-br-int 2/none: (patch: peer=patch-br-int-to-provnet-5a77e708-5c1f-48dc-acdf-21e66d8e3be7)
      br-int:
        br-int 65534/1: (internal)
        ovn-juju-c-0 2/4: (geneve: csum=true, key=flow, remote_ip=10.5.0.191)
        patch-br-int-to-provnet-5a77e708-5c1f-48dc-acdf-21e66d8e3be7 1/none: (patch: peer=patch-provnet-5a77e708-5c1f-48dc-acdf-21e66d8e3be7-to-br-int)
        tap1ba6b6f4-14 5/7: (system)
        tapcd9fefdb-00 3/5: (system)
        tapd0382b73-e0 4/6: (system)
        tapf1b85533-30 6/8: (system)
    
    root@juju-c40d4b-ovn-6:~# ovs-vsctl show
    2849984e-7c3c-4390-b6f0-2cb47c757ca0
        Manager "ptcp:6640:127.0.0.1"
            is_connected: true
        Bridge br-int
            fail_mode: secure
            datapath_type: system
            Port tapcd9fefdb-00
                Interface tapcd9fefdb-00
            Port tap1ba6b6f4-14
                Interface tap1ba6b6f4-14
            Port tapd0382b73-e0
                Interface tapd0382b73-e0
            Port tapf1b85533-30
                Interface tapf1b85533-30
            Port br-int
                Interface br-int
                    type: internal
            Port ovn-juju-c-0
                Interface ovn-juju-c-0
                    type: geneve
                    options: {csum="true", key=flow, remote_ip="10.5.0.191"}
            Port patch-br-int-to-provnet-5a77e708-5c1f-48dc-acdf-21e66d8e3be7
                Interface patch-br-int-to-provnet-5a77e708-5c1f-48dc-acdf-21e66d8e3be7
                    type: patch
                    options: {peer=patch-provnet-5a77e708-5c1f-48dc-acdf-21e66d8e3be7-to-br-int}
        Bridge br-data
            fail_mode: standalone
            datapath_type: system
            Port patch-provnet-5a77e708-5c1f-48dc-acdf-21e66d8e3be7-to-br-int
                Interface patch-provnet-5a77e708-5c1f-48dc-acdf-21e66d8e3be7-to-br-int
                    type: patch
                    options: {peer=patch-br-int-to-provnet-5a77e708-5c1f-48dc-acdf-21e66d8e3be7}
            Port br-data
                Interface br-data
                    type: internal
            Port ens8
                Interface ens8
                    type: system
        ovs_version: "2.13.1"
    
    当ssh into i1，在l3上会看到：
    tcp      6 426771 ESTABLISHED src=10.5.0.8 dst=10.5.150.115 sport=39746 dport=22 src=192.168.21.161 dst=10.5.0.8 sport=22 dport=39746 [ASSURED] mark=0 zone=1 use=1
    tcp      6 431995 ESTABLISHED src=10.5.0.8 dst=10.5.150.115 sport=39904 dport=22 src=192.168.21.161 dst=10.5.0.8 sport=22 dport=39904 [ASSURED] mark=0 zone=1 use=1
    
    当从i1上运行"nc -uvz 10.5.0.2 53", 在L3上会看到
    root@juju-c40d4b-ovn-6:~# conntrack -L |grep 192.168.21.161
    conntrack v1.4.4 (conntrack-tools): 52 flow entries have been shown.
    udp      17 25 src=192.168.21.161 dst=10.5.0.2 sport=55185 dport=53 [UNREPLIED] src=10.5.0.2 dst=10.5.150.115 sport=53 dport=55185 mark=0 zone=2 use=1
    udp      17 6 src=192.168.21.161 dst=10.5.0.2 sport=36199 dport=53 [UNREPLIED] src=10.5.0.2 dst=10.5.150.115 sport=53 dport=36199 mark=0 zone=2 use=1
    
    在SouthBound DB master上看到了ovn逻辑流是：
    ubuntu@zhhuabj-bastion:~/stsstack-bundles/openstack$ juju ssh ovn-central/2 -- sudo -s
    root@juju-c40d4b-ovn-9:~# ovn-trace --detail neutron-b2f023b1-4a05-4443-b334-cf47e90a1567 'inport == "13d0a59c-e25d-48f5-af68-ca18dbbf139d" && eth.src == fa:16:3e:54:36:ad && ip4.dst == 10.5.0.2'
    # ip,reg14=0x3,vlan_tci=0x0000,dl_src=fa:16:3e:54:36:ad,dl_dst=00:00:00:00:00:00,nw_src=0.0.0.0,nw_dst=10.5.0.2,nw_proto=0,nw_tos=0,nw_ecn=0,nw_ttl=0
    ingress(dp="private", inport="13d0a5")
    --------------------------------------
     0. ls_in_port_sec_l2 (ovn-northd.c:4516): inport == "13d0a5" && eth.src == {fa:16:3e:54:36:ad}, priority 50, uuid 620c23f4
        next;
     1. ls_in_port_sec_ip (ovn-northd.c:4225): inport == "13d0a5" && eth.src == fa:16:3e:54:36:ad && ip, priority 80, uuid 47d5e0e8
    
    在L3上调试openflow
    sudo snap install ovs-stat
    sudo snap connect ovs-stat:openvswitch
    sudo snap connect ovs-stat:network-control
    #sudo snap connect ovs-stat:removable-media
    #ovs-stat -p /tmp/results --tree ./sosreport-015 --openstack  #don't use sudo
    ovs-stat -p /tmp/results --tree --openstack
    sudo ls /tmp/snap.ovs-stat/tmp/results
    ovs-stat -p /tmp/results --host juju-c40d4b-ovn-6 --query ""
    root@juju-c40d4b-ovn-6:~# ovs-stat -p /tmp/results --host juju-c40d4b-ovn-6 --query "ofproto-trace.port tapf1b85533-30"
    [arp]
    no source ips found - skipping
    [icmp]
    no source ips found - skipping
    [dhcp]
    sudo ovs-appctl ofproto/trace br-int udp,in_port=6,dl_src=12:6b:c5:a6:f5:47,dl_dst=ff:ff:ff:ff:ff:ff,nw_src=0.0.0.0,nw_dst=255.255.255.255,udp_src=68,udp_dst=67
    [vm-to-vm]
    sudo ovs-appctl ofproto/trace br-int in_port=6,tcp,dl_src=12:6b:c5:a6:f5:47,dl_dst=MAC_OF_REMOTE_INSTANCE
    sudo ovs-appctl ofproto/trace br-int in_port=6,dl_vlan=,dl_src=12:6b:c5:a6:f5:47,dl_dst=MAC_OF_REMOTE_INSTANCE
    
    在L3节点上看到的openflow flow是：
    root@juju-c40d4b-ovn-6:~# ovs-ofctl dump-flows br-int |grep 192.168.21.161
     cookie=0xe51693a7, duration=86438.109s, table=14, n_packets=22223, n_bytes=1532774, idle_age=33, hard_age=65534, priority=100,ip,reg14=0x1,metadata=0x2,nw_dst=10.5.150.115 actions=ct(commit,table=15,zone=NXM_NX_REG11[0..15],nat(dst=192.168.21.161))
     cookie=0x1043afc5, duration=86542.428s, table=21, n_packets=1, n_bytes=42, idle_age=65534, hard_age=65534, priority=50,arp,metadata=0x3,arp_tpa=192.168.21.161,arp_op=1 actions=move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],mod_dl_src:fa:16:3e:54:36:ad,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],load:0xfa163e5436ad->NXM_NX_ARP_SHA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],load:0xc0a815a1->NXM_OF_ARP_SPA[],move:NXM_NX_REG14[]->NXM_NX_REG15[],load:0x1->NXM_NX_REG10[0],resubmit(,32)
     cookie=0xd56bfbc4, duration=86438.109s, table=40, n_packets=53762, n_bytes=13079808, idle_age=33, hard_age=65534, priority=100,ip,reg15=0x1,metadata=0x2,nw_src=192.168.21.161 actions=ct(table=41,zone=NXM_NX_REG11[0..15],nat)
     cookie=0xf8bdd8ec, duration=86438.109s, table=41, n_packets=16596, n_bytes=1295012, idle_age=34, hard_age=65534, priority=161,ip,reg15=0x1,metadata=0x2,nw_src=192.168.21.161 actions=ct(commit,table=42,zone=NXM_NX_REG12[0..15],nat(src=10.5.150.115))
     cookie=0x0, duration=85847.305s, table=44, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=2002,ct_state=-new+est-rpl+trk,ct_label=0/0x1,ip,metadata=0x3,nw_src=192.168.21.161 actions=conjunction(9,1/2)
     cookie=0x0, duration=85847.305s, table=44, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=2002,ct_state=-new+est-rpl+trk,ct_label=0x1/0x1,ip,metadata=0x4,nw_src=192.168.21.161 actions=conjunction(3,1/2)
     cookie=0x0, duration=85847.306s, table=44, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=2002,ct_state=-new+est-rpl+trk,ct_label=0x1/0x1,ip,metadata=0x3,nw_src=192.168.21.161 actions=conjunction(7,1/2)
     cookie=0x0, duration=85847.305s, table=44, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=2002,ct_state=-new+est-rpl+trk,ct_label=0/0x1,ip,metadata=0x4,nw_src=192.168.21.161 actions=conjunction(5,1/2)
     cookie=0x0, duration=85847.306s, table=44, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=2002,ct_state=+new-est+trk,ip,metadata=0x3,nw_src=192.168.21.161 actions=conjunction(6,1/2)
     cookie=0x0, duration=85847.305s, table=44, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=2002,ct_state=+new-est+trk,ip,metadata=0x4,nw_src=192.168.21.161 actions=conjunction(2,1/2)
     cookie=0x0, duration=85847.305s, table=44, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=2002,ct_state=-trk,ip,metadata=0x4,nw_src=192.168.21.161 actions=conjunction(4,1/2)
     cookie=0x0, duration=85847.305s, table=44, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=2002,ct_state=-trk,ip,metadata=0x3,nw_src=192.168.21.161 actions=conjunction(8,1/2)

## 附录 - NB与SB表 ##

root@juju-c40d4b-ovn-8:~# ovn-nbctl show
    switch 38364b40-19f0-473f-8878-da50b652cc67 (neutron-1d7749fd-90c9-4f31-ada4-50f1845ca32e) (aka ext_net)
        port 7a9ae9c2-3dd5-498b-9c4d-09a101fc3120
            type: localport
            addresses: ["fa:16:3e:b0:67:15"]
        port 304cfe5a-d25c-41aa-bfbe-9ba60c7248c2
            type: router
            router-port: lrp-304cfe5a-d25c-41aa-bfbe-9ba60c7248c2
        port provnet-5a77e708-5c1f-48dc-acdf-21e66d8e3be7
            type: localnet
            addresses: ["unknown"]
    switch c6cc6d66-91af-4613-87aa-cbd770d8040d (neutron-b2f023b1-4a05-4443-b334-cf47e90a1567) (aka private)
        port abe38147-7909-4708-ad02-d478e62e7ff1
            type: router
            router-port: lrp-abe38147-7909-4708-ad02-d478e62e7ff1
        port 13d0a59c-e25d-48f5-af68-ca18dbbf139d
            addresses: ["fa:16:3e:54:36:ad 192.168.21.161"]
        port 0a2c4125-791c-4824-837a-1a940e78673a
            type: localport
            addresses: ["fa:16:3e:2d:ce:27 192.168.21.2"]
        port cd9fefdb-00f0-4efd-950b-84ba32788571
            addresses: ["fa:16:3e:78:2f:34 192.168.21.3"]
    switch c53d2b6f-6721-4b97-bb8e-c62df9bd952b (neutron-1c537bdd-5633-4263-a364-b14cecd4e92d) (aka private2)
        port 1ba6b6f4-140b-4bb7-a0fd-6d880cda47ff
            addresses: ["fa:16:3e:d2:1d:ec 192.168.22.47"]
        port b236113a-86e2-4d69-8de8-f1086cc17a7b
            type: router
            router-port: lrp-b236113a-86e2-4d69-8de8-f1086cc17a7b
        port 3999ba6e-7f80-499b-8dd8-fa87d0f4a63e
            type: localport
            addresses: ["fa:16:3e:76:aa:1e 192.168.22.2"]
    router 89a12d3b-28ad-466b-97d6-971c669aee44 (neutron-1a4bf8d2-c885-4af8-8b9c-061c7b27fa69) (aka provider-router)
        port lrp-abe38147-7909-4708-ad02-d478e62e7ff1
            mac: "fa:16:3e:22:d6:67"
            networks: ["192.168.21.1/24"]
        port lrp-b236113a-86e2-4d69-8de8-f1086cc17a7b
            mac: "fa:16:3e:93:b1:62"
            networks: ["192.168.22.1/24"]
        port lrp-304cfe5a-d25c-41aa-bfbe-9ba60c7248c2
            mac: "fa:16:3e:50:aa:2a"
            networks: ["10.5.152.46/16"]
            gateway chassis: [juju-c40d4b-ovn-6.cloud.sts]
        nat 6196ba5f-568b-486e-b7d6-add825d2f8f9
            external ip: "10.5.152.46"
            logical ip: "192.168.21.0/24"
            type: "snat"
        nat b1e5878e-95f0-45f7-b3a2-232b550be281
            external ip: "10.5.152.46"
            logical ip: "192.168.22.0/24"
        nat cb82abf5-55b5-4731-aa24-b993ac4621d9
            external ip: "10.5.150.115"
            logical ip: "192.168.21.161"
            type: "dnat_and_snat"
    
    root@juju-c40d4b-ovn-8:~# ovn-sbctl show
    Chassis juju-c40d4b-ovn-13.cloud.sts
        hostname: juju-c40d4b-ovn-13.cloud.sts
        Encap geneve
            ip: "10.5.0.191"
            options: {csum="true"}
        Port_Binding "13d0a59c-e25d-48f5-af68-ca18dbbf139d"
    Chassis juju-c40d4b-ovn-6.cloud.sts
        hostname: juju-c40d4b-ovn-6.cloud.sts
        Encap geneve
            ip: "10.5.0.178"
            options: {csum="true"}
        Port_Binding cr-lrp-304cfe5a-d25c-41aa-bfbe-9ba60c7248c2
        Port_Binding "cd9fefdb-00f0-4efd-950b-84ba32788571"
        Port_Binding "1ba6b6f4-140b-4bb7-a0fd-6d880cda47ff"

## OVN Southbound DB CLI ##

#https://blog.csdn.net/zhengmx100/article/details/75426710
    # https://docs.openstack.org/networking-ovn/ocata/refarch/refarch.html
    ovn-nbctl list Logical_Switch
    ovn-nbctl list Logical_Switch_Port
    ovn-nbctl list ACL
    ovn-nbctl list Address_Set
    ovn-nbctl list Logical_Router
    ovn-nbctl list Logical_Router_Port
    ovn-sbctl list Chassis
    ovn-sbctl list Encap
    ovn-nbctl list Address_Set
    ovn-sbctl lflow-list
    ovn-sbctl list Multicast_Group
    ovn-sbctl list Datapath_Binding
    ovn-sbctl list Port_Binding
    ovn-sbctl list MAC_Binding
    ovn-nbctl list Gateway_Chassis
    ovn-nbctl list dhcp_options
    
    ovn-nbctl show
    ovn-sbctl show
    
    # ovn-sbctl show
    Chassis juju-e28e17-ovn2-8.cloud.sts
        hostname: juju-e28e17-ovn2-8.cloud.sts
        Encap geneve
            ip: "10.5.2.173"
            options: {csum="true"}
        Port_Binding "bdbe0b57-eb1e-41d5-a06d-b2fd3050fa4d"
        Port_Binding cr-lrp-a3076445-eb22-411b-ad68-2672c0abcaa3
    
    #从它找到Port_Binding对应的chassis
    # ovn-sbctl list Port_Binding cr-lrp-a3076445-eb22-411b-ad68-2672c0abcaa3
    _uuid               : c689e404-2534-40b7-92b0-0a0cb306c458
    chassis             : 3ffd5c8b-86c1-498d-9128-84289a1e832a
    datapath            : 2ec13388-171d-4719-9b6f-39a755a68f60
    encap               : []
    external_ids        : {}
    gateway_chassis     : []
    ha_chassis_group    : beeb76fd-aa54-43f6-aeed-2c1d71acf2b4
    logical_port        : cr-lrp-a3076445-eb22-411b-ad68-2672c0abcaa3
    mac                 : ["fa:16:3e:1c:06:c6 10.5.151.201/16"]
    nat_addresses       : []
    options             : {distributed-port=lrp-a3076445-eb22-411b-ad68-2672c0abcaa3}
    parent_port         : []
    tag                 : []
    tunnel_key          : 2
    type                : chassisredirect
    virtual_parent      : []
    
    # 再从chassis找到encaps
    # ovn-sbctl list Chassis
    _uuid               : 3ffd5c8b-86c1-498d-9128-84289a1e832a
    encaps              : [0406d6d8-b1d2-45b4-8461-22b661bcb7dd]
    external_ids        : {datapath-type=system, iface-types="erspan,geneve,gre,internal,ip6erspan,ip6gre,lisp,patch,stt,system,tap,vxlan", is-interconn="false", neutron-metadata-proxy-networks="e2ed122c-9442-4116-9793-5c2323f6de29", "neutron:liveness_check_at"="2021-04-01T10:09:44.354244+00:00", "neutron:metadata_liveness_check_at"="2021-04-01T10:09:45.089358+00:00", "neutron:ovn-metadata-id"="6e1038ee-52f4-4bdf-afb9-0e220696176c", "neutron:ovn-metadata-sb-cfg"="571", ovn-bridge-mappings="physnet1:br-data", ovn-chassis-mac-mappings="", ovn-cms-options=enable-chassis-as-gw}
    hostname            : juju-e28e17-ovn2-8.cloud.sts
    name                : juju-e28e17-ovn2-8.cloud.sts
    nb_cfg              : 571
    transport_zones     : []
    vtep_logical_switches: []
    
    #从Encap找到tunnel IP
    # ovn-sbctl list Encap
    _uuid               : 0406d6d8-b1d2-45b4-8461-22b661bcb7dd
    chassis_name        : juju-e28e17-ovn2-8.cloud.sts
    ip                  : "10.5.2.173"
    options             : {csum="true"}
    type                : geneve
    
    # ovn-sbctl list Datapath_Binding
    _uuid               : e2ed122c-9442-4116-9793-5c2323f6de29
    external_ids        : {logical-switch="eb7511fa-6746-4981-9ddc-6e1c73e9a7ee", name=neutron-b3187a23-b54e-4cc0-bc6d-4caabdb02b0c, name2=private}
    tunnel_key          : 3
    _uuid               : 2ec13388-171d-4719-9b6f-39a755a68f60
    external_ids        : {logical-router="8bbdfe13-4266-432e-a1b9-aae2dacf6ebd", name=neutron-0bb8206c-96e5-4fd8-a6ad-66e591620496, name2=provider-router}
    tunnel_key          : 2
    _uuid               : a00e41b6-d46b-4ce4-a0f9-ddfcca7c241c
    external_ids        : {logical-switch="375943d6-4b12-44c2-907d-ca59e8a3915d", name=neutron-bbc0362f-44ac-4bab-accf-613c68c6fb66, name2=ext_net}
    
    ovn-sbctl lflow-list |grep icmp

## 如何找到dhcp在哪个host上呢？ ##

## 用OVN实现OpenStack原理以流表分析 ##

1, 每个Node(Chassis）上创建br-data2
    
    ovs-vsctl --may-exist add-br br-data2
    #ovs-vsctl --may-exist add-port br-data2 eth2
    ovs-vsctl set open . external-ids:ovn-bridge-mappings=physnet1:br-data,physnet2:br-data2
    
    2, 东西向的两个Subnet(Switch)和vRouter(Logical Router)
    
    ovn-nbctl ls-add sw0
    ovn-nbctl lsp-add sw0 sw0-port1
    ovn-nbctl lsp-set-addresses sw0-port1 "00:00:01:00:00:03 10.0.0.3"
    
    ovn-nbctl ls-add sw1
    ovn-nbctl lsp-add sw1 sw1-port1
    ovn-nbctl lsp-set-addresses sw1-port1 "00:00:02:00:00:03 20.0.0.3"
    
    ovn-nbctl lr-add lr0
    # Connect sw0 to lr0
    ovn-nbctl lrp-add lr0 lr0-sw0 00:00:00:00:ff:01 10.0.0.1/24
    ovn-nbctl lsp-add sw0 sw0-lr0
    ovn-nbctl lsp-set-type sw0-lr0 router
    ovn-nbctl lsp-set-addresses sw0-lr0 router
    ovn-nbctl lsp-set-options sw0-lr0 router-port=lr0-sw0
    # Connect sw1 to lr0
    ovn-nbctl lrp-add lr0 lr0-sw1 00:00:00:00:ff:02 20.0.0.1/24
    ovn-nbctl lsp-add sw1 sw1-lr0
    ovn-nbctl lsp-set-type sw1-lr0 router
    ovn-nbctl lsp-set-addresses sw1-lr0 router
    ovn-nbctl lsp-set-options sw1-lr0 router-port=lr0-sw1
    ovn-nbctl show
    
    3, 南北向流量，br-ex(public)，并在vRouter(lr0)中设置外部IP（即将lr0与public关联）
       这里network_name为physnet2, 且localnet port定义在switch上意味着运行在Gateway Chassis (Centralized L3)
       上的ovn-controller将在br-int与br-data2之前创建patch port
    
    ovn-nbctl ls-add public
    # Create a localnet port
    ovn-nbctl lsp-add public ln-public
    ovn-nbctl lsp-set-type ln-public localnet
    ovn-nbctl lsp-set-addresses ln-public unknown
    ovn-nbctl lsp-set-options ln-public network_name=physnet2
    
    # 这个vRouter中的外部接口(lr0-public)也应该部署在L3上，它在将流量经patch port转给br-ex前需SNAT
    # 并且当有人要访问外部地址172.168.0.200时,Gateway Chassis也需响应ARP reply
    ovn-nbctl lrp-add lr0 lr0-public 00:00:20:20:12:13 172.168.0.200/24
    ovn-nbctl lsp-add public public-lr0
    ovn-nbctl lsp-set-type public-lr0 router
    ovn-nbctl lsp-set-addresses public-lr0 router
    ovn-nbctl lsp-set-options public-lr0 router-port=lr0-public
    
    有两个方法schedule Gateway router port(lr0-public):
    a, non_HA, 只有single Gateway Chassis, eg: schedule到ovn-controller
       ovn-nbctl set logical_router_port lr0-public options:redirect-chassis=juju-e28e17-ovn2-8.cloud.sts
       # ovn-sbctl show |grep lr0-public
          Port_Binding cr-lr0-public
    b, HA, 有多个Gateway Chassis, 一个挂了，另外高优先级的能起来，类似于VRRP
       ovn-nbctl lrp-set-gateway-chassis lr0-public juju-e28e17-ovn2-8.cloud.sts 20
       ovn-nbctl lrp-set-gateway-chassis lr0-public controller-1 15
       ovn-nbctl lrp-set-gateway-chassis lr0-public controller-2 10
       #ovn-nbctl lrp-del-gateway-chassis lr0-public controller-1
       "ovn-nbctl list gateway_chassis"能看到有多个Gateway Chassis
       "ovn-nbctl list logical_router_port lr0-public"也能看到多个Gateway Chassis
       对于这个HA模式，OVN使用BFD(Bidirectional Forwarding Detection)协议，它配置BFD在tunnel ports上,
       当一个Gateway Chassis上的Distributed gateway port挂了的话，其他的Gateway Chassis上都能检测到
       它的type是chassisredirect
       # ovn-sbctl list Port_Binding cr-lr0-public |grep type
       type                : chassisredirect
    
    4, 南北向出虚机流量的流表如下：
    a, vrouter上的内部网关(lr0-sw0)与外部网关(lr0-public)将有下列流表(逻辑流)：
     table=0 (lr_in_admission ), priority=50 , match=(eth.dst == 00:00:00:00:ff:01 && inport == "lr0-sw0"), action=(next;)
     table=7 (lr_in_ip_routing ), priority=49 , match=(ip4.dst == 172.168.0.0/24), action=(ip.ttl--; reg0 = ip4.dst; reg1 = 172.168.0.200; eth.src = 00:00:20:20:12:13; outport = "lr0-public"; flags.loopback = 1; next;)
     table=9 (lr_in_gw_redirect ), priority=50 , match=(outport == "lr0-public"), action=(outport = "cr-lr0-public"; next;)
    
    b, 假如 cr-lr0-public被schedule在controller-0上，包应该走tunnel port
     table=32, priority=100,reg15=0x4,metadata=0x3 actions=load:0x3->NXM_NX_TUN_ID[0..23],set_field:0x4->tun_metadata0,move:NXM_NX_REG14[0..14]->NXM_NX_TUN_METADATA0[16..30],output:ovn-cont-0
    
    c, controller-0 chassis收到tunnel port的包，并把它转到lr0
    table=0, priority=100,in_port="ovn-comp-0" actions=move:NXM_NX_TUN_ID[0..23]->OXM_OF_METADATA[0..23],move:NXM_NX_TUN_METADATA0[16..30]->NXM_NX_REG14[0..14],move:NXM_NX_TUN_METADATA0[0..15]->NXM_NX_REG15[0..15],resubmit(,33)
    
    d, 接着是SNAT
    table=1 (lr_out_snat ), priority=25 , match=(ip && ip4.src == 10.0.0.0/24 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat(172.168.0.200);)
    
    e, 发到vRouter的外网网关lr0-public port
    table=3 (lr_out_delivery ), priority=100 , match=(outport == "lr0-public"), action=(output;)
    
    f, 包将localnet port(patch port)进provider bridge(br-ex)并最终到达目的地
    
    
    5, 南北向进虚机流量的流表如下：
    
    在controller-0 chassis上：
    
    a, 物理网关收到包，走br-ex，再经local net port(patch port)进vRouter(lr0)
    
     table=0,priority=100,in_port="patch-br-int-to",dl_vlan=0 actions=strip_vlan,load:0x1->NXM_NX_REG13[],load:0x7->NXM_NX_REG11[],load:0x8->NXM_NX_REG12[],load:0x4->OXM_OF_METADATA[],load:0x2->NXM_NX_REG14[],resubmit(,8)
    
    b, DNAT(UnSNAT): 172.168.0.200 -> 10.0.0.3
    
    table=0 (lr_in_admission ), priority=50 , match=(eth.dst == 00:00:20:20:12:13 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(next;)
     table=3 (lr_in_unsnat ), priority=100 , match=(ip && ip4.dst == 172.168.0.200 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
    
    c, 10.0.0.3属于Switch sw0, vRouter(lr0)需将流量导到lr0-sw0
    
    table=7 (lr_in_ip_routing ), priority=49 , match=(ip4.dst == 10.0.0.0/24), action=(ip.ttl--; reg0 = ip4.dst; reg1 = 10.0.0.1; eth.src = 00:00:00:00:ff:01; outport = "lr0-sw0"; flags.loopback = 1; next;)
    
    e, The ingress pipeline of sw0 is run and the packet is sent to compute-0 via the tunnel port because OVN knows that sw0-port1 resides on compute-0.
    
    在compute-0 Chassic上：
    
    f, compute-0 receives the traffic on the tunnel port and sends the traffic to the egress pipeline of logical switch sw0, In the egress pipeline, the packet is delivered to sw0-port1.

## 从ovnmeta ns中ping同台节点虚机的流表 ##

例如tapbdbe0b57-eb/fe:16:3e:ec:f5:9a对应的是虚机，而tape2ed122c-90是ovnmeta ns里的tap
    ip netns exec ovnmeta-e2ed122c-9442-4116-9793-5c2323f6de29 ip addr show 
    tape2ed122c-91@if12  -> tape2ed122c-90@if2
    
    # ovs-vsctl -- --columns=name,ofport list Interface tape2ed122c-90
    name                : tape2ed122c-90
    ofport              : 3
    
    ovs-vsctl -- --columns=name,ofport list Interface tapbdbe0b57-eb
    name                : tapbdbe0b57-eb
    ofport              : 2
    
    # ovs-ofctl -O OpenFlow13 dump-flows br-int |grep 'in_port=3'
     cookie=0x3384108f, duration=38141.438s, table=0, n_packets=0, n_bytes=0, priority=100,in_port=3 actions=set_field:0x9->reg13,set_field:0x1->reg11,set_field:0x6->reg12,set_field:0x3->metadata,set_field:0x1->reg14,resubmit(,8)
    
    Flow: in_port=3,vlan_tci=0x0000,dl_src=fe:16:3e:ec:f5:9a,dl_dst=12:d6:31:fe:59:a8,dl_type=0x0000
    
    bridge("br-int")
    ----------------
     - in_port=3, priority 100, cookie 0x3384108f
        set_field:0x9->reg13
        set_field:0x1->reg11
        set_field:0x6->reg12
        set_field:0x3->metadata
        set_field:0x1->reg14
        resubmit(,8)
     - reg14=0x1,metadata=0x3, priority 50, cookie 0x3c9c4889
        resubmit(,9)
     - metadata=0x3, priority 0, cookie 0x6475678d
        resubmit(,10)
     - metadata=0x3, priority 0, cookie 0x9593deb1
        resubmit(,11)
     - metadata=0x3, priority 0, cookie 0x4db625a0
        resubmit(,12)
     - metadata=0x3, priority 0, cookie 0x395e746e
        resubmit(,13)
     - metadata=0x3, priority 0, cookie 0x57defbe5
        resubmit(,14)
     - metadata=0x3, priority 0, cookie 0x32c7898e
        resubmit(,15)
     - metadata=0x3, priority 0, cookie 0xf06ee7a1
        resubmit(,16)
     - metadata=0x3, priority 0, cookie 0x351d116c
        resubmit(,17)
     - metadata=0x3, priority 0, cookie 0x3f2d1d9e
        resubmit(,18)
     - metadata=0x3, priority 0, cookie 0x42601c1a
        resubmit(,19)
     - metadata=0x3, priority 0, cookie 0x2b6f92a0
        resubmit(,20)
     - metadata=0x3, priority 0, cookie 0xabefbd0d
        resubmit(,21)
     - metadata=0x3, priority 0, cookie 0x4854b450
        resubmit(,22)
     - metadata=0x3, priority 0, cookie 0xf346b65d
        resubmit(,23)
     - metadata=0x3, priority 0, cookie 0xb2d4f8c9
        resubmit(,24)
     - metadata=0x3, priority 0, cookie 0x49d3f6be
        resubmit(,25)
     - metadata=0x3, priority 0, cookie 0x5bd4e34b
        resubmit(,26)
     - metadata=0x3, priority 0, cookie 0x8e0cea9
        resubmit(,27)
     - No match.
        drop
    Final flow: reg11=0x1,reg12=0x6,reg13=0x9,reg14=0x1,metadata=0x3,in_port=3,vlan_tci=0x0000,dl_src=fe:16:3e:ec:f5:9a,dl_dst=12:d6:31:fe:59:a8,dl_type=0x0000
    Megaflow: recirc_id=0,ct_state=-new-est-rel-rpl-inv-trk,ct_label=0/0x1,eth,in_port=3,dl_src=fe:16:3e:ec:f5:9a,dl_dst=12:d6:31:fe:59:a8,dl_type=0x0000
    Datapath actions: drop

## 从ovnmeta ns中ping同台节点虚机的逻辑流表 ##

https://blog.csdn.net/zhengmx100/article/details/78140948  
流表是单个机器的(流表由table组成，table包含flow,flow又有priority, match, action)，逻辑流表是多台机器编排的(逻辑流表下发到每台机器的ovn-controller，而ovn-controller知道根据当前的物理环境即本地端口如何到达其他机器而转换成流表）。  
例如创建一个logical switch sw0, 再创建两个port。sw0将有两个pipeline (ingress pipeline与egress pipline), 两个port port在相同主机与不同相同主机所走的pipeline不一样，如下：  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3F1cWk5OQ_size_16_color_FFFFFF_t_70]  
![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3F1cWk5OQ_size_16_color_FFFFFF_t_70 1]  
逻辑流表就是类似于neutron的port等这些高层概念，它的port只是网络中的一个元素并不限定在某台节点上。可以使用"ovn-sbctl lflow-list"命令来查看完整的logical flow。理解logical flow最好的方式就是使用ovn-trace命令。ovn-trace能够让你看到OVN对一个包是怎么处理的。

ovn-trace DATAPATH MICROFLOW 
    例如，还是用ovnmeta ns(192.168.21.2/12:d6:31:fe:59:a8)中来ping VM(192.168.21.18/fe:16:3e:ec:f5:9a)
    switch eb7511fa-6746-4981-9ddc-6e1c73e9a7ee (neutron-b3187a23-b54e-4cc0-bc6d-4caabdb02b0c) (aka private)
        port bdbe0b57-eb1e-41d5-a06d-b2fd3050fa4d
            addresses: ["fa:16:3e:ec:f5:9a 192.168.21.18"]
        port a6194e3b-6de5-4334-beb7-a898760488d0
            type: localport
            addresses: ["fa:16:3e:de:68:05 192.168.21.2"]
        ...
    
    # ovn-trace --summary private 'inport == "a6194e3b-6de5-4334-beb7-a898760488d0" && eth.src == 12:d6:31:fe:59:a8 && eth.dst == fe:16:3e:ec:f5:9a'
    # reg14=0x1,vlan_tci=0x0000,dl_src=12:d6:31:fe:59:a8,dl_dst=fe:16:3e:ec:f5:9a,dl_type=0x0000
    ingress(dp="private", inport="a6194e") {
        next;
    };
    root@juju-e28e17-ovn2-10:/home/ubuntu# ovn-trace --detail private 'inport == "a6194e3b-6de5-4334-beb7-a898760488d0" && eth.src == 12:d6:31:fe:59:a8 && eth.dst == fe:16:3e:ec:f5:9a'
    # reg14=0x1,vlan_tci=0x0000,dl_src=12:d6:31:fe:59:a8,dl_dst=fe:16:3e:ec:f5:9a,dl_type=0x0000
    
    ingress(dp="private", inport="a6194e")
    --------------------------------------
     0. ls_in_port_sec_l2 (ovn-northd.c:4514): inport == "a6194e", priority 50, uuid 3c9c4889
        next;
    19. ls_in_l2_lkup: no match (implicit drop)

## Possible bugs ##

\[1\] https://bugs.launchpad.net/charm-ovn-chassis/+bug/1907686  
\[2\] https://bugs.launchpad.net/charm-neutron-api/+bug/1921986

## Reference ##

\[1\] https://networkop.co.uk/blog/2016/12/10/ovn-part2/  
\[2\] https://numans.blog/2018/11/30/how-to-create-an-open-virtual-network-distributed-gateway-router/

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3F1cWk5OQ_size_16_color_FFFFFF_t_70_pic_center]: /images/20221023/80d9bd2b3bd246b7b9d02a203314911f.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3F1cWk5OQ_size_16_color_FFFFFF_t_70_pic_center 1]: /images/20221023/aece5e768b584acdbd3eecf1de58f125.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3F1cWk5OQ_size_16_color_FFFFFF_t_70_pic_center 2]: /images/20221023/a35d10ce6eee4adf9f6b38d77ef1b07b.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3F1cWk5OQ_size_16_color_FFFFFF_t_70]: /images/20221023/6a7596ca0afd403fac10863e19efe02e.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3F1cWk5OQ_size_16_color_FFFFFF_t_70 1]: /images/20221023/bf12658c62944289b41ec5aafc883f66.png