dive into openstack ovn (by quqi99)

旧城等待, 2022-11-05 08:37 302阅读 0赞

作者:张华 发表于:2021-03-04
版权声明:可以任意转载,转载时请务必以超链接形式标明文章原始出处和作者信息及本版权声明

前一篇基础是:Play with OVN - https://blog.csdn.net/quqi99/article/details/103194137
这一篇将主要讲openstack如何来使用ovn的。
在这里插入图片描述

测试环境

下面将搭建一下类似的测试环境。

  1. ./generate-bundle.sh --name ovn --series bionic --release ussuri --ovn --vault --create-model --run
  2. juju add-unit nova-compute
  3. ./configure
  4. source novarc
  5. neutron net-create private2 --provider:network_type geneve --provider:segmentation_id 1012
  6. neutron subnet-create --gateway 192.168.22.1 private2 192.168.22.0/24 --enable_dhcp=True --name private_subnet2
  7. ROUTER_ID=$(neutron router-list |grep ' provider-router ' |awk '{print $2}')
  8. SUBNET_ID=$(neutron subnet-list |grep '192.168.22.0/24' |awk '{print $2}')
  9. neutron router-interface-add $ROUTER_ID $SUBNET_ID
  10. nova hypervisor-list
  11. openstack server create --wait --image bionic --flavor m1.small --key-name testkey --nic net-id=$(openstack net show private -f value -c id) --availability-zone=nova:juju-c40d4b-ovn-13.cloud.sts i1
  12. ./tools/float_all.sh
  13. ./tools/sec_groups.sh
  14. openstack server create --wait --image cirros2 --flavor m1.small --key-name testkey --nic net-id=$(openstack net show private -f value -c id) --availability-zone=nova:juju-c40d4b-ovn-6.cloud.sts i2
  15. openstack server create --wait --image cirros2 --flavor m1.small --key-name testkey --nic net-id=$(openstack net show private2 -f value -c id) --availability-zone=nova:juju-c40d4b-ovn-6.cloud.sts i3
  16. $ nova list
  17. +--------------------------------------+------+--------+------------+-------------+--------------------------------------+
  18. | ID | Name | Status | Task State | Power State | Networks |
  19. +--------------------------------------+------+--------+------------+-------------+--------------------------------------+
  20. | 82c89129-0335-4e33-b117-be940a7020d4 | i1 | ACTIVE | - | Running | private=192.168.21.161, 10.5.150.115 |
  21. | 74641e74-3401-44f9-8d7d-bef3ea0fdb92 | i2 | ACTIVE | - | Running | private=192.168.21.3 |
  22. | 37f7c5c0-844c-4d8d-ad95-aa29b7418dc0 | i3 | ACTIVE | - | Running | private2=192.168.22.47 |
  23. +--------------------------------------+------+--------+------------+-------------+--------------------------------------+

OVN Northbound DB与Neutron的概念映射

Neutron中有Network, Subnet, Router, Port的概念,OVN Northbound DB中也有对应的逻辑概念: Switch=Neutron Subnet, Port=Neutron Port, Distributed Router=Neutron DVR Router, Gateway Router=Neutron Centralized L3, Port=Neutron Port

举例查看OVN Northbound DB中和L3 NAT相关的数据

  1. #run in compute node
  2. # ovs-vsctl get open . external_ids
  3. {hostname=juju-c40d4b-ovn-6.cloud.sts, ovn-bridge-mappings="physnet1:br-data", ovn-cms-options=enable-chassis-as-gw, ovn-encap-ip="10.5.0.178", ovn-encap-type=geneve, ovn-remote="ssl:10.5.2.178:6642,ssl:10.5.1.220:6642,ssl:10.5.1.157:6642", rundir="/var/run/openvswitch", system-id=juju-c40d4b-ovn-6.cloud.sts}
  4. export SB=$(sudo ovs-vsctl get open . external_ids:ovn-remote | sed -e 's/\"//g')
  5. export NB=$(sudo ovs-vsctl get open . external_ids:ovn-remote | sed -e 's/\"//g' | sed -e 's/6642/6641/g')
  6. #从所有计算节点上的ovn-controller中所出用于中心化l3的那个(搜索enable-chassis-as-gw得知是juju-c40d4b-ovn-6.cloud.sts(uuid=f8004279-14d2-48fd-8b6a-f025706fa8a8)
  7. #run in ovnnb_db master and ovnsb_db master
  8. juju ssh ovn-central/1 -- sudo -s
  9. root@juju-c40d4b-ovn-8:~# ovn-sbctl list chassis
  10. _uuid : add1028c-e19c-4f23-8795-a0c64f16fdcd
  11. encaps : [7417dd02-d2d5-45bb-88b9-3dafe07c92c6]
  12. external_ids : {datapath-type=system, iface-types="erspan,geneve,gre,internal,ip6erspan,ip6gre,lisp,patch,stt,system,tap,vxlan", is-interconn="false", neutron-metadata-proxy-networks="d0382b73-eb07-4314-a803-b957662f618c", "neutron:liveness_check_at"="2021-03-04T10:34:37.869388+00:00", "neutron:metadata_liveness_check_at"="2021-03-04T10:34:38.163481+00:00", "neutron:ovn-metadata-id"="4849b5a6-3134-4ca2-9fea-38c24aef6121", "neutron:ovn-metadata-sb-cfg"="578", ovn-bridge-mappings="", ovn-chassis-mac-mappings="", ovn-cms-options=""}
  13. hostname : juju-c40d4b-ovn-13.cloud.sts
  14. name : juju-c40d4b-ovn-13.cloud.sts
  15. nb_cfg : 578
  16. transport_zones : []
  17. vtep_logical_switches: []
  18. _uuid : f8004279-14d2-48fd-8b6a-f025706fa8a8
  19. encaps : [50fde256-2d20-4e4d-aa6e-c7838edee407]
  20. external_ids : {datapath-type=system, iface-types="erspan,geneve,gre,internal,ip6erspan,ip6gre,lisp,patch,stt,system,tap,vxlan", is-interconn="false", neutron-metadata-proxy-networks="d0382b73-eb07-4314-a803-b957662f618c,f1b85533-3f78-4f44-9785-996e725bb3bf", "neutron:liveness_check_at"="2021-03-04T10:34:37.322922+00:00", "neutron:metadata_liveness_check_at"="2021-03-04T10:34:37.602977+00:00", "neutron:ovn-metadata-id"="9dd62f6d-c41b-42e2-b424-7d0bbf0902ea", "neutron:ovn-metadata-sb-cfg"="578", ovn-bridge-mappings="physnet1:br-data", ovn-chassis-mac-mappings="", ovn-cms-options=enable-chassis-as-gw}
  21. hostname : juju-c40d4b-ovn-6.cloud.sts
  22. name : juju-c40d4b-ovn-6.cloud.sts
  23. nb_cfg : 578
  24. transport_zones : []
  25. vtep_logical_switches: []
  26. #或者直接找到它l3 ovn-controller
  27. root@juju-c40d4b-ovn-8:~# ovn-nbctl list Gateway_Chassis
  28. _uuid : 5de561d4-77e5-467d-8b69-ec064e949d8c
  29. chassis_name : juju-c40d4b-ovn-6.cloud.sts
  30. external_ids : {}
  31. name : lrp-304cfe5a-d25c-41aa-bfbe-9ba60c7248c2_juju-c40d4b-ovn-6.cloud.sts
  32. options : {}
  33. priority : 1
  34. #找到neutron router的external_gateway_info和routerid=1a4bf8d2-c885-4af8-8b9c-061c7b27fa69
  35. $ openstack router show provider-router --fit-width |grep external_gateway_info
  36. | external_gateway_info | {"network_id": "1d7749fd-90c9-4f31-ada4-50f1845ca32e", "external_fixed_ips": [{"subnet_id": "4009f18b-eb09-4b74-a0ac-ce29537838a3", "ip_address": "10.5.152.46"}], "enable_snat": true}
  37. #找到与此neutron router对应的ovn router
  38. root@juju-c40d4b-ovn-8:~# ovn-nbctl find Logical_Router name=neutron-1a4bf8d2-c885-4af8-8b9c-061c7b27fa69
  39. _uuid : 89a12d3b-28ad-466b-97d6-971c669aee44
  40. enabled : true
  41. external_ids : {"neutron:availability_zone_hints"="", "neutron:gw_port_id"="304cfe5a-d25c-41aa-bfbe-9ba60c7248c2", "neutron:revision_number"="5", "neutron:router_name"=provider-router}
  42. load_balancer : []
  43. name : neutron-1a4bf8d2-c885-4af8-8b9c-061c7b27fa69
  44. nat : [6196ba5f-568b-486e-b7d6-add825d2f8f9, b1e5878e-95f0-45f7-b3a2-232b550be281, cb82abf5-55b5-4731-aa24-b993ac4621d9]
  45. options : {}
  46. policies : []
  47. ports : [3aa3c7ce-ec0f-4d3e-9a3b-15ae7f750622, f964314d-ac73-4dda-a940-9c6910850b34, fb90ec4c-ac5e-4415-96e8-28ea18e53205]
  48. static_routes : [cd814869-e8a0-4dde-9730-362d5d83a1d0]
  49. #从OVN NAT northbound表中验证SNAT
  50. root@juju-c40d4b-ovn-8:~# ovn-nbctl find NAT type=snat
  51. _uuid : b1e5878e-95f0-45f7-b3a2-232b550be281
  52. external_ids : {}
  53. external_ip : "10.5.152.46"
  54. external_mac : []
  55. logical_ip : "192.168.22.0/24"
  56. logical_port : []
  57. options : {}
  58. type : snat
  59. _uuid : 6196ba5f-568b-486e-b7d6-add825d2f8f9
  60. external_ids : {}
  61. external_ip : "10.5.152.46"
  62. external_mac : []
  63. logical_ip : "192.168.21.0/24"
  64. logical_port : []
  65. options : {}
  66. type : snat
  67. root@juju-c40d4b-ovn-8:~# ovn-nbctl find NAT type=dnat_and_snat
  68. _uuid : cb82abf5-55b5-4731-aa24-b993ac4621d9
  69. external_ids : {"neutron:fip_external_mac"="fa:16:3e:b6:11:c5", "neutron:fip_id"="9bf3a29c-e7fe-4dca-8c59-a8809ae87db9", "neutron:fip_port_id"="13d0a59c-e25d-48f5-af68-ca18dbbf139d", "neutron:revision_number"="2", "neutron:router_name"=neutron-1a4bf8d2-c885-4af8-8b9c-061c7b27fa69}
  70. external_ip : "10.5.150.115"
  71. external_mac : []
  72. logical_ip : "192.168.21.161"
  73. logical_port : "13d0a59c-e25d-48f5-af68-ca18dbbf139d"
  74. options : {}
  75. type : dnat_and_snat
  76. root@juju-c40d4b-ovn-8:~# ovn-nbctl lr-nat-list neutron-1a4bf8d2-c885-4af8-8b9c-061c7b27fa69
  77. TYPE EXTERNAL_IP LOGICAL_IP EXTERNAL_MAC LOGICAL_PORT
  78. dnat_and_snat 10.5.150.115 192.168.21.161
  79. snat 10.5.152.46 192.168.21.0/24
  80. snat 10.5.152.46 192.168.22.0/24

OVN Southbound DB - L2 Logical Flow (同网段大二层)

在这里插入图片描述

  1. $ openstack port list
  2. +--------------------------------------+------+-------------------+-------------------------------------------------------------------------------+--------+
  3. | ID | Name | MAC Address | Fixed IP Addresses | Status |
  4. +--------------------------------------+------+-------------------+-------------------------------------------------------------------------------+--------+
  5. | 0a2c4125-791c-4824-837a-1a940e78673a | | fa:16:3e:2d:ce:27 | ip_address='192.168.21.2', subnet_id='a1607cfe-3fa8-40a1-891e-776801f90342' | DOWN |
  6. | 13d0a59c-e25d-48f5-af68-ca18dbbf139d | | fa:16:3e:54:36:ad | ip_address='192.168.21.161', subnet_id='a1607cfe-3fa8-40a1-891e-776801f90342' | ACTIVE |
  7. | 1ba6b6f4-140b-4bb7-a0fd-6d880cda47ff | | fa:16:3e:d2:1d:ec | ip_address='192.168.22.47', subnet_id='450d6660-b862-4908-b501-4c8533211b23' | ACTIVE |
  8. | 304cfe5a-d25c-41aa-bfbe-9ba60c7248c2 | | fa:16:3e:50:aa:2a | ip_address='10.5.152.46', subnet_id='4009f18b-eb09-4b74-a0ac-ce29537838a3' | ACTIVE |
  9. | 3999ba6e-7f80-499b-8dd8-fa87d0f4a63e | | fa:16:3e:76:aa:1e | ip_address='192.168.22.2', subnet_id='450d6660-b862-4908-b501-4c8533211b23' | DOWN |
  10. | 7a9ae9c2-3dd5-498b-9c4d-09a101fc3120 | | fa:16:3e:b0:67:15 | | DOWN |
  11. | abe38147-7909-4708-ad02-d478e62e7ff1 | | fa:16:3e:22:d6:67 | ip_address='192.168.21.1', subnet_id='a1607cfe-3fa8-40a1-891e-776801f90342' | ACTIVE |
  12. | b236113a-86e2-4d69-8de8-f1086cc17a7b | | fa:16:3e:93:b1:62 | ip_address='192.168.22.1', subnet_id='450d6660-b862-4908-b501-4c8533211b23' | ACTIVE |
  13. | cd9fefdb-00f0-4efd-950b-84ba32788571 | | fa:16:3e:78:2f:34 | ip_address='192.168.21.3', subnet_id='a1607cfe-3fa8-40a1-891e-776801f90342' | ACTIVE |
  14. | eb40953f-28f3-46f4-a28a-50786d1090b5 | | fa:16:3e:b6:11:c5 | ip_address='10.5.150.115', subnet_id='4009f18b-eb09-4b74-a0ac-ce29537838a3' | N/A |
  15. +--------------------------------------+------+-------------------+-------------------------------------------------------------------------------+--------+

对于vm1=i1(192.168.21.161)访问vm3=i3(192.168.21.3)

  1. $ openstack port list |grep -E '192.168.21.3|192.168.21.161'
  2. | 13d0a59c-e25d-48f5-af68-ca18dbbf139d | | fa:16:3e:54:36:ad | ip_address='192.168.21.161', subnet_id='a1607cfe-3fa8-40a1-891e-776801f90342' | ACTIVE |
  3. | cd9fefdb-00f0-4efd-950b-84ba32788571 | | fa:16:3e:78:2f:34 | ip_address='192.168.21.3', subnet_id='a1607cfe-3fa8-40a1-891e-776801f90342' | ACTIVE |

1, port security只是对从虚机进来的包检查它的IP与MAC是否对应

  1. root@juju-c40d4b-ovn-8:~# ovn-sbctl lflow-list |grep inport |grep 13d0a59c-e25d-48f5-af68-ca18dbbf139d |grep -E 'table=0'
  2. table=0 (ls_in_port_sec_l2 ), priority=50 , match=(inport == "13d0a59c-e25d-48f5-af68-ca18dbbf139d" && eth.src == {fa:16:3e:54:36:ad}), action=(next;)

2,

接下来的看这篇文章 -
OpenStack SDN With OVN (Part 2) - Network Engineering Analysis
https://networkop.co.uk/blog/2016/12/10/ovn-part2/

OVN Southbound DB - L3 Logical Flow(南北)

在这里插入图片描述
见 -  https://networkop.co.uk/blog/2016/12/10/ovn-part2/

OVN Southbound DB - ovn controller Logical Flow (不同网段东西)

通过ovn-trace调试OVN Sourthbound DB逻辑流

ovn-trace能用来帮助调试或者理解上面的ovn南向逻辑流

  1. root@juju-c40d4b-ovn-8:~# ovn-sbctl lflow-list |grep -i datapath
  2. Datapath: "neutron-1a4bf8d2-c885-4af8-8b9c-061c7b27fa69" aka "provider-router" (587cfb5a-2797-405f-83f0-385211e2ad78) Pipeline: ingress
  3. Datapath: "neutron-1a4bf8d2-c885-4af8-8b9c-061c7b27fa69" aka "provider-router" (587cfb5a-2797-405f-83f0-385211e2ad78) Pipeline: egress
  4. Datapath: "neutron-1d7749fd-90c9-4f31-ada4-50f1845ca32e" aka "ext_net" (74359f20-4009-4c7e-afd8-3f3dd2423b77) Pipeline: ingress
  5. Datapath: "neutron-1d7749fd-90c9-4f31-ada4-50f1845ca32e" aka "ext_net" (74359f20-4009-4c7e-afd8-3f3dd2423b77) Pipeline: egress
  6. Datapath: "neutron-b2f023b1-4a05-4443-b334-cf47e90a1567" aka "private" (d0382b73-eb07-4314-a803-b957662f618c) Pipeline: ingress
  7. Datapath: "neutron-b2f023b1-4a05-4443-b334-cf47e90a1567" aka "private" (d0382b73-eb07-4314-a803-b957662f618c) Pipeline: egress
  8. Datapath: "neutron-1c537bdd-5633-4263-a364-b14cecd4e92d" aka "private2" (f1b85533-3f78-4f44-9785-996e725bb3bf) Pipeline: ingress
  9. Datapath: "neutron-1c537bdd-5633-4263-a364-b14cecd4e92d" aka "private2" (f1b85533-3f78-4f44-9785-996e725bb3bf) Pipeline: egress
  10. root@juju-c40d4b-ovn-8:~# ovn-nbctl show c6cc6d66-91af-4613-87aa-cbd770d8040d
  11. switch c6cc6d66-91af-4613-87aa-cbd770d8040d (neutron-b2f023b1-4a05-4443-b334-cf47e90a1567) (aka private)
  12. port abe38147-7909-4708-ad02-d478e62e7ff1
  13. type: router
  14. router-port: lrp-abe38147-7909-4708-ad02-d478e62e7ff1
  15. port 13d0a59c-e25d-48f5-af68-ca18dbbf139d
  16. addresses: ["fa:16:3e:54:36:ad 192.168.21.161"]
  17. port 0a2c4125-791c-4824-837a-1a940e78673a
  18. type: localport
  19. addresses: ["fa:16:3e:2d:ce:27 192.168.21.2"]
  20. port cd9fefdb-00f0-4efd-950b-84ba32788571
  21. addresses: ["fa:16:3e:78:2f:34 192.168.21.3"]

例如上面的192.168.21.161如何访问192.168.21.3的.

  1. root@juju-c40d4b-ovn-8:~# ovn-trace --minimal neutron-b2f023b1-4a05-4443-b334-cf47e90a1567 'inport == "13d0a59c-e25d-48f5-af68-ca18dbbf139d" && eth.src == fa:16:3e:54:36:ad && eth.dst == fa:16:3e:78:2f:34'
  2. # reg14=0x3,vlan_tci=0x0000,dl_src=fa:16:3e:54:36:ad,dl_dst=fa:16:3e:78:2f:34,dl_type=0x0000
  3. output("cd9fef");
  4. root@juju-c40d4b-ovn-8:~# ovn-trace neutron-b2f023b1-4a05-4443-b334-cf47e90a1567 'inport == "13d0a59c-e25d-48f5-af68-ca18dbbf139d" && eth.src == fa:16:3e:54:36:ad && eth.dst == fa:16:3e:78:2f:34'
  5. # reg14=0x3,vlan_tci=0x0000,dl_src=fa:16:3e:54:36:ad,dl_dst=fa:16:3e:78:2f:34,dl_type=0x0000
  6. ingress(dp="private", inport="13d0a5")
  7. --------------------------------------
  8. 0. ls_in_port_sec_l2 (ovn-northd.c:4516): inport == "13d0a5" && eth.src == {fa:16:3e:54:36:ad}, priority 50, uuid 620c23f4
  9. next;
  10. 19. ls_in_l2_lkup (ovn-northd.c:6779): eth.dst == fa:16:3e:78:2f:34, priority 50, uuid cdf396b9
  11. outport = "cd9fef";
  12. output;
  13. egress(dp="private", inport="13d0a5", outport="cd9fef")
  14. -------------------------------------------------------
  15. 9. ls_out_port_sec_l2 (ovn-northd.c:4582): outport == "cd9fef" && eth.dst == {fa:16:3e:78:2f:34}, priority 50, uuid b48eb374
  16. output;
  17. /* output to "cd9fef", type "" */
  18. root@juju-c40d4b-ovn-8:~# ovn-trace --summary neutron-b2f023b1-4a05-4443-b334-cf47e90a1567 'inport == "13d0a59c-e25d-48f5-af68-ca18dbbf139d" && eth.src == fa:16:3e:54:36:ad && eth.dst == fa:16:3e:78:2f:34'
  19. # reg14=0x3,vlan_tci=0x0000,dl_src=fa:16:3e:54:36:ad,dl_dst=fa:16:3e:78:2f:34,dl_type=0x0000
  20. ingress(dp="private", inport="13d0a5") {
  21. next;
  22. outport = "cd9fef";
  23. output;
  24. egress(dp="private", inport="13d0a5", outport="cd9fef") {
  25. output;
  26. /* output to "cd9fef", type "" */;
  27. };
  28. };

再例如如何访问8.8.8.8

  1. root@juju-c40d4b-ovn-8:~# ovn-trace --detail neutron-b2f023b1-4a05-4443-b334-cf47e90a1567 'inport == "13d0a59c-e25d-48f5-af68-ca18dbbf139d" && eth.src == fa:16:3e:54:36:ad && ip4.dst == 8.8.8.8'
  2. # ip,reg14=0x3,vlan_tci=0x0000,dl_src=fa:16:3e:54:36:ad,dl_dst=00:00:00:00:00:00,nw_src=0.0.0.0,nw_dst=8.8.8.8,nw_proto=0,nw_tos=0,nw_ecn=0,nw_ttl=0
  3. ingress(dp="private", inport="13d0a5")
  4. --------------------------------------
  5. 0. ls_in_port_sec_l2 (ovn-northd.c:4516): inport == "13d0a5" && eth.src == {fa:16:3e:54:36:ad}, priority 50, uuid 620c23f4
  6. next;
  7. 1. ls_in_port_sec_ip (ovn-northd.c:4225): inport == "13d0a5" && eth.src == fa:16:3e:54:36:ad && ip, priority 80, uuid 47d5e0e8
  8. drop;

计算节点上的openflow flow

上面的都是OVN南向DB的逻辑流,直接到计算节点上的openflow通过“ovs-ofctl dump-flows br-int”查看,可通过“ovs-appctl ofproto/trace”来调试(ovn Logical Flow流过ovn-trace调试),见:https://blog.russellbryant.net/2016/11/11/ovn-logical-flows-and-ovn-trace/

如:
sudo ovs-appctl ofproto/trace br-int in_port=6,arp,arp_spa=192.168.21.7,dl_src=fa:16:3e:c4:58:9c
可使用ovs-stat snap工具来方便生成ovs-appctl辅助命令,见: https://blog.csdn.net/quqi99/article/details/111831695

实操 - 虚机访问外部UDP服务调试流表

  1. 找到了centralized l3juju-c40d4b-ovn-6.cloud.sts, 
  2. root@juju-c40d4b-ovn-6:~# sudo ovs-appctl dpif/show
  3. system@ovs-system: hit:243081 missed:5389
  4. br-data:
  5. br-data 65534/2: (internal)
  6. ens8 1/3: (system)
  7. patch-provnet-5a77e708-5c1f-48dc-acdf-21e66d8e3be7-to-br-int 2/none: (patch: peer=patch-br-int-to-provnet-5a77e708-5c1f-48dc-acdf-21e66d8e3be7)
  8. br-int:
  9. br-int 65534/1: (internal)
  10. ovn-juju-c-0 2/4: (geneve: csum=true, key=flow, remote_ip=10.5.0.191)
  11. patch-br-int-to-provnet-5a77e708-5c1f-48dc-acdf-21e66d8e3be7 1/none: (patch: peer=patch-provnet-5a77e708-5c1f-48dc-acdf-21e66d8e3be7-to-br-int)
  12. tap1ba6b6f4-14 5/7: (system)
  13. tapcd9fefdb-00 3/5: (system)
  14. tapd0382b73-e0 4/6: (system)
  15. tapf1b85533-30 6/8: (system)
  16. root@juju-c40d4b-ovn-6:~# ovs-vsctl show
  17. 2849984e-7c3c-4390-b6f0-2cb47c757ca0
  18. Manager "ptcp:6640:127.0.0.1"
  19. is_connected: true
  20. Bridge br-int
  21. fail_mode: secure
  22. datapath_type: system
  23. Port tapcd9fefdb-00
  24. Interface tapcd9fefdb-00
  25. Port tap1ba6b6f4-14
  26. Interface tap1ba6b6f4-14
  27. Port tapd0382b73-e0
  28. Interface tapd0382b73-e0
  29. Port tapf1b85533-30
  30. Interface tapf1b85533-30
  31. Port br-int
  32. Interface br-int
  33. type: internal
  34. Port ovn-juju-c-0
  35. Interface ovn-juju-c-0
  36. type: geneve
  37. options: {csum="true", key=flow, remote_ip="10.5.0.191"}
  38. Port patch-br-int-to-provnet-5a77e708-5c1f-48dc-acdf-21e66d8e3be7
  39. Interface patch-br-int-to-provnet-5a77e708-5c1f-48dc-acdf-21e66d8e3be7
  40. type: patch
  41. options: {peer=patch-provnet-5a77e708-5c1f-48dc-acdf-21e66d8e3be7-to-br-int}
  42. Bridge br-data
  43. fail_mode: standalone
  44. datapath_type: system
  45. Port patch-provnet-5a77e708-5c1f-48dc-acdf-21e66d8e3be7-to-br-int
  46. Interface patch-provnet-5a77e708-5c1f-48dc-acdf-21e66d8e3be7-to-br-int
  47. type: patch
  48. options: {peer=patch-br-int-to-provnet-5a77e708-5c1f-48dc-acdf-21e66d8e3be7}
  49. Port br-data
  50. Interface br-data
  51. type: internal
  52. Port ens8
  53. Interface ens8
  54. type: system
  55. ovs_version: "2.13.1"
  56. ssh into i1,在l3上会看到:
  57. tcp 6 426771 ESTABLISHED src=10.5.0.8 dst=10.5.150.115 sport=39746 dport=22 src=192.168.21.161 dst=10.5.0.8 sport=22 dport=39746 [ASSURED] mark=0 zone=1 use=1
  58. tcp 6 431995 ESTABLISHED src=10.5.0.8 dst=10.5.150.115 sport=39904 dport=22 src=192.168.21.161 dst=10.5.0.8 sport=22 dport=39904 [ASSURED] mark=0 zone=1 use=1
  59. 当从i1上运行"nc -uvz 10.5.0.2 53", L3上会看到
  60. root@juju-c40d4b-ovn-6:~# conntrack -L |grep 192.168.21.161
  61. conntrack v1.4.4 (conntrack-tools): 52 flow entries have been shown.
  62. udp 17 25 src=192.168.21.161 dst=10.5.0.2 sport=55185 dport=53 [UNREPLIED] src=10.5.0.2 dst=10.5.150.115 sport=53 dport=55185 mark=0 zone=2 use=1
  63. udp 17 6 src=192.168.21.161 dst=10.5.0.2 sport=36199 dport=53 [UNREPLIED] src=10.5.0.2 dst=10.5.150.115 sport=53 dport=36199 mark=0 zone=2 use=1
  64. SouthBound DB master上看到了ovn逻辑流是:
  65. ubuntu@zhhuabj-bastion:~/stsstack-bundles/openstack$ juju ssh ovn-central/2 -- sudo -s
  66. root@juju-c40d4b-ovn-9:~# ovn-trace --detail neutron-b2f023b1-4a05-4443-b334-cf47e90a1567 'inport == "13d0a59c-e25d-48f5-af68-ca18dbbf139d" && eth.src == fa:16:3e:54:36:ad && ip4.dst == 10.5.0.2'
  67. # ip,reg14=0x3,vlan_tci=0x0000,dl_src=fa:16:3e:54:36:ad,dl_dst=00:00:00:00:00:00,nw_src=0.0.0.0,nw_dst=10.5.0.2,nw_proto=0,nw_tos=0,nw_ecn=0,nw_ttl=0
  68. ingress(dp="private", inport="13d0a5")
  69. --------------------------------------
  70. 0. ls_in_port_sec_l2 (ovn-northd.c:4516): inport == "13d0a5" && eth.src == {fa:16:3e:54:36:ad}, priority 50, uuid 620c23f4
  71. next;
  72. 1. ls_in_port_sec_ip (ovn-northd.c:4225): inport == "13d0a5" && eth.src == fa:16:3e:54:36:ad && ip, priority 80, uuid 47d5e0e8
  73. L3上调试openflow
  74. sudo snap install ovs-stat
  75. sudo snap connect ovs-stat:openvswitch
  76. sudo snap connect ovs-stat:network-control
  77. #sudo snap connect ovs-stat:removable-media
  78. #ovs-stat -p /tmp/results --tree ./sosreport-015 --openstack #don't use sudo
  79. ovs-stat -p /tmp/results --tree --openstack
  80. sudo ls /tmp/snap.ovs-stat/tmp/results
  81. ovs-stat -p /tmp/results --host juju-c40d4b-ovn-6 --query ""
  82. root@juju-c40d4b-ovn-6:~# ovs-stat -p /tmp/results --host juju-c40d4b-ovn-6 --query "ofproto-trace.port tapf1b85533-30"
  83. [arp]
  84. no source ips found - skipping
  85. [icmp]
  86. no source ips found - skipping
  87. [dhcp]
  88. sudo ovs-appctl ofproto/trace br-int udp,in_port=6,dl_src=12:6b:c5:a6:f5:47,dl_dst=ff:ff:ff:ff:ff:ff,nw_src=0.0.0.0,nw_dst=255.255.255.255,udp_src=68,udp_dst=67
  89. [vm-to-vm]
  90. sudo ovs-appctl ofproto/trace br-int in_port=6,tcp,dl_src=12:6b:c5:a6:f5:47,dl_dst=MAC_OF_REMOTE_INSTANCE
  91. sudo ovs-appctl ofproto/trace br-int in_port=6,dl_vlan=,dl_src=12:6b:c5:a6:f5:47,dl_dst=MAC_OF_REMOTE_INSTANCE
  92. L3节点上看到的openflow flow是:
  93. root@juju-c40d4b-ovn-6:~# ovs-ofctl dump-flows br-int |grep 192.168.21.161
  94. cookie=0xe51693a7, duration=86438.109s, table=14, n_packets=22223, n_bytes=1532774, idle_age=33, hard_age=65534, priority=100,ip,reg14=0x1,metadata=0x2,nw_dst=10.5.150.115 actions=ct(commit,table=15,zone=NXM_NX_REG11[0..15],nat(dst=192.168.21.161))
  95. cookie=0x1043afc5, duration=86542.428s, table=21, n_packets=1, n_bytes=42, idle_age=65534, hard_age=65534, priority=50,arp,metadata=0x3,arp_tpa=192.168.21.161,arp_op=1 actions=move:NXM_OF_ETH_SRC[]->NXM_OF_ETH_DST[],mod_dl_src:fa:16:3e:54:36:ad,load:0x2->NXM_OF_ARP_OP[],move:NXM_NX_ARP_SHA[]->NXM_NX_ARP_THA[],load:0xfa163e5436ad->NXM_NX_ARP_SHA[],move:NXM_OF_ARP_SPA[]->NXM_OF_ARP_TPA[],load:0xc0a815a1->NXM_OF_ARP_SPA[],move:NXM_NX_REG14[]->NXM_NX_REG15[],load:0x1->NXM_NX_REG10[0],resubmit(,32)
  96. cookie=0xd56bfbc4, duration=86438.109s, table=40, n_packets=53762, n_bytes=13079808, idle_age=33, hard_age=65534, priority=100,ip,reg15=0x1,metadata=0x2,nw_src=192.168.21.161 actions=ct(table=41,zone=NXM_NX_REG11[0..15],nat)
  97. cookie=0xf8bdd8ec, duration=86438.109s, table=41, n_packets=16596, n_bytes=1295012, idle_age=34, hard_age=65534, priority=161,ip,reg15=0x1,metadata=0x2,nw_src=192.168.21.161 actions=ct(commit,table=42,zone=NXM_NX_REG12[0..15],nat(src=10.5.150.115))
  98. cookie=0x0, duration=85847.305s, table=44, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=2002,ct_state=-new+est-rpl+trk,ct_label=0/0x1,ip,metadata=0x3,nw_src=192.168.21.161 actions=conjunction(9,1/2)
  99. cookie=0x0, duration=85847.305s, table=44, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=2002,ct_state=-new+est-rpl+trk,ct_label=0x1/0x1,ip,metadata=0x4,nw_src=192.168.21.161 actions=conjunction(3,1/2)
  100. cookie=0x0, duration=85847.306s, table=44, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=2002,ct_state=-new+est-rpl+trk,ct_label=0x1/0x1,ip,metadata=0x3,nw_src=192.168.21.161 actions=conjunction(7,1/2)
  101. cookie=0x0, duration=85847.305s, table=44, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=2002,ct_state=-new+est-rpl+trk,ct_label=0/0x1,ip,metadata=0x4,nw_src=192.168.21.161 actions=conjunction(5,1/2)
  102. cookie=0x0, duration=85847.306s, table=44, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=2002,ct_state=+new-est+trk,ip,metadata=0x3,nw_src=192.168.21.161 actions=conjunction(6,1/2)
  103. cookie=0x0, duration=85847.305s, table=44, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=2002,ct_state=+new-est+trk,ip,metadata=0x4,nw_src=192.168.21.161 actions=conjunction(2,1/2)
  104. cookie=0x0, duration=85847.305s, table=44, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=2002,ct_state=-trk,ip,metadata=0x4,nw_src=192.168.21.161 actions=conjunction(4,1/2)
  105. cookie=0x0, duration=85847.305s, table=44, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=2002,ct_state=-trk,ip,metadata=0x3,nw_src=192.168.21.161 actions=conjunction(8,1/2)

附录 - NB与SB表

  1. root@juju-c40d4b-ovn-8:~# ovn-nbctl show
  2. switch 38364b40-19f0-473f-8878-da50b652cc67 (neutron-1d7749fd-90c9-4f31-ada4-50f1845ca32e) (aka ext_net)
  3. port 7a9ae9c2-3dd5-498b-9c4d-09a101fc3120
  4. type: localport
  5. addresses: ["fa:16:3e:b0:67:15"]
  6. port 304cfe5a-d25c-41aa-bfbe-9ba60c7248c2
  7. type: router
  8. router-port: lrp-304cfe5a-d25c-41aa-bfbe-9ba60c7248c2
  9. port provnet-5a77e708-5c1f-48dc-acdf-21e66d8e3be7
  10. type: localnet
  11. addresses: ["unknown"]
  12. switch c6cc6d66-91af-4613-87aa-cbd770d8040d (neutron-b2f023b1-4a05-4443-b334-cf47e90a1567) (aka private)
  13. port abe38147-7909-4708-ad02-d478e62e7ff1
  14. type: router
  15. router-port: lrp-abe38147-7909-4708-ad02-d478e62e7ff1
  16. port 13d0a59c-e25d-48f5-af68-ca18dbbf139d
  17. addresses: ["fa:16:3e:54:36:ad 192.168.21.161"]
  18. port 0a2c4125-791c-4824-837a-1a940e78673a
  19. type: localport
  20. addresses: ["fa:16:3e:2d:ce:27 192.168.21.2"]
  21. port cd9fefdb-00f0-4efd-950b-84ba32788571
  22. addresses: ["fa:16:3e:78:2f:34 192.168.21.3"]
  23. switch c53d2b6f-6721-4b97-bb8e-c62df9bd952b (neutron-1c537bdd-5633-4263-a364-b14cecd4e92d) (aka private2)
  24. port 1ba6b6f4-140b-4bb7-a0fd-6d880cda47ff
  25. addresses: ["fa:16:3e:d2:1d:ec 192.168.22.47"]
  26. port b236113a-86e2-4d69-8de8-f1086cc17a7b
  27. type: router
  28. router-port: lrp-b236113a-86e2-4d69-8de8-f1086cc17a7b
  29. port 3999ba6e-7f80-499b-8dd8-fa87d0f4a63e
  30. type: localport
  31. addresses: ["fa:16:3e:76:aa:1e 192.168.22.2"]
  32. router 89a12d3b-28ad-466b-97d6-971c669aee44 (neutron-1a4bf8d2-c885-4af8-8b9c-061c7b27fa69) (aka provider-router)
  33. port lrp-abe38147-7909-4708-ad02-d478e62e7ff1
  34. mac: "fa:16:3e:22:d6:67"
  35. networks: ["192.168.21.1/24"]
  36. port lrp-b236113a-86e2-4d69-8de8-f1086cc17a7b
  37. mac: "fa:16:3e:93:b1:62"
  38. networks: ["192.168.22.1/24"]
  39. port lrp-304cfe5a-d25c-41aa-bfbe-9ba60c7248c2
  40. mac: "fa:16:3e:50:aa:2a"
  41. networks: ["10.5.152.46/16"]
  42. gateway chassis: [juju-c40d4b-ovn-6.cloud.sts]
  43. nat 6196ba5f-568b-486e-b7d6-add825d2f8f9
  44. external ip: "10.5.152.46"
  45. logical ip: "192.168.21.0/24"
  46. type: "snat"
  47. nat b1e5878e-95f0-45f7-b3a2-232b550be281
  48. external ip: "10.5.152.46"
  49. logical ip: "192.168.22.0/24"
  50. nat cb82abf5-55b5-4731-aa24-b993ac4621d9
  51. external ip: "10.5.150.115"
  52. logical ip: "192.168.21.161"
  53. type: "dnat_and_snat"
  54. root@juju-c40d4b-ovn-8:~# ovn-sbctl show
  55. Chassis juju-c40d4b-ovn-13.cloud.sts
  56. hostname: juju-c40d4b-ovn-13.cloud.sts
  57. Encap geneve
  58. ip: "10.5.0.191"
  59. options: {csum="true"}
  60. Port_Binding "13d0a59c-e25d-48f5-af68-ca18dbbf139d"
  61. Chassis juju-c40d4b-ovn-6.cloud.sts
  62. hostname: juju-c40d4b-ovn-6.cloud.sts
  63. Encap geneve
  64. ip: "10.5.0.178"
  65. options: {csum="true"}
  66. Port_Binding cr-lrp-304cfe5a-d25c-41aa-bfbe-9ba60c7248c2
  67. Port_Binding "cd9fefdb-00f0-4efd-950b-84ba32788571"
  68. Port_Binding "1ba6b6f4-140b-4bb7-a0fd-6d880cda47ff"

OVN Southbound DB CLI

  1. #https://blog.csdn.net/zhengmx100/article/details/75426710
  2. # https://docs.openstack.org/networking-ovn/ocata/refarch/refarch.html
  3. ovn-nbctl list Logical_Switch
  4. ovn-nbctl list Logical_Switch_Port
  5. ovn-nbctl list ACL
  6. ovn-nbctl list Address_Set
  7. ovn-nbctl list Logical_Router
  8. ovn-nbctl list Logical_Router_Port
  9. ovn-sbctl list Chassis
  10. ovn-sbctl list Encap
  11. ovn-nbctl list Address_Set
  12. ovn-sbctl lflow-list
  13. ovn-sbctl list Multicast_Group
  14. ovn-sbctl list Datapath_Binding
  15. ovn-sbctl list Port_Binding
  16. ovn-sbctl list MAC_Binding
  17. ovn-nbctl list Gateway_Chassis
  18. ovn-nbctl list dhcp_options
  19. ovn-nbctl show
  20. ovn-sbctl show
  21. # ovn-sbctl show
  22. Chassis juju-e28e17-ovn2-8.cloud.sts
  23. hostname: juju-e28e17-ovn2-8.cloud.sts
  24. Encap geneve
  25. ip: "10.5.2.173"
  26. options: {csum="true"}
  27. Port_Binding "bdbe0b57-eb1e-41d5-a06d-b2fd3050fa4d"
  28. Port_Binding cr-lrp-a3076445-eb22-411b-ad68-2672c0abcaa3
  29. #从它找到Port_Binding对应的chassis
  30. # ovn-sbctl list Port_Binding cr-lrp-a3076445-eb22-411b-ad68-2672c0abcaa3
  31. _uuid : c689e404-2534-40b7-92b0-0a0cb306c458
  32. chassis : 3ffd5c8b-86c1-498d-9128-84289a1e832a
  33. datapath : 2ec13388-171d-4719-9b6f-39a755a68f60
  34. encap : []
  35. external_ids : {}
  36. gateway_chassis : []
  37. ha_chassis_group : beeb76fd-aa54-43f6-aeed-2c1d71acf2b4
  38. logical_port : cr-lrp-a3076445-eb22-411b-ad68-2672c0abcaa3
  39. mac : ["fa:16:3e:1c:06:c6 10.5.151.201/16"]
  40. nat_addresses : []
  41. options : {distributed-port=lrp-a3076445-eb22-411b-ad68-2672c0abcaa3}
  42. parent_port : []
  43. tag : []
  44. tunnel_key : 2
  45. type : chassisredirect
  46. virtual_parent : []
  47. # 再从chassis找到encaps
  48. # ovn-sbctl list Chassis
  49. _uuid : 3ffd5c8b-86c1-498d-9128-84289a1e832a
  50. encaps : [0406d6d8-b1d2-45b4-8461-22b661bcb7dd]
  51. external_ids : {datapath-type=system, iface-types="erspan,geneve,gre,internal,ip6erspan,ip6gre,lisp,patch,stt,system,tap,vxlan", is-interconn="false", neutron-metadata-proxy-networks="e2ed122c-9442-4116-9793-5c2323f6de29", "neutron:liveness_check_at"="2021-04-01T10:09:44.354244+00:00", "neutron:metadata_liveness_check_at"="2021-04-01T10:09:45.089358+00:00", "neutron:ovn-metadata-id"="6e1038ee-52f4-4bdf-afb9-0e220696176c", "neutron:ovn-metadata-sb-cfg"="571", ovn-bridge-mappings="physnet1:br-data", ovn-chassis-mac-mappings="", ovn-cms-options=enable-chassis-as-gw}
  52. hostname : juju-e28e17-ovn2-8.cloud.sts
  53. name : juju-e28e17-ovn2-8.cloud.sts
  54. nb_cfg : 571
  55. transport_zones : []
  56. vtep_logical_switches: []
  57. #从Encap找到tunnel IP
  58. # ovn-sbctl list Encap
  59. _uuid : 0406d6d8-b1d2-45b4-8461-22b661bcb7dd
  60. chassis_name : juju-e28e17-ovn2-8.cloud.sts
  61. ip : "10.5.2.173"
  62. options : {csum="true"}
  63. type : geneve
  64. # ovn-sbctl list Datapath_Binding
  65. _uuid : e2ed122c-9442-4116-9793-5c2323f6de29
  66. external_ids : {logical-switch="eb7511fa-6746-4981-9ddc-6e1c73e9a7ee", name=neutron-b3187a23-b54e-4cc0-bc6d-4caabdb02b0c, name2=private}
  67. tunnel_key : 3
  68. _uuid : 2ec13388-171d-4719-9b6f-39a755a68f60
  69. external_ids : {logical-router="8bbdfe13-4266-432e-a1b9-aae2dacf6ebd", name=neutron-0bb8206c-96e5-4fd8-a6ad-66e591620496, name2=provider-router}
  70. tunnel_key : 2
  71. _uuid : a00e41b6-d46b-4ce4-a0f9-ddfcca7c241c
  72. external_ids : {logical-switch="375943d6-4b12-44c2-907d-ca59e8a3915d", name=neutron-bbc0362f-44ac-4bab-accf-613c68c6fb66, name2=ext_net}
  73. ovn-sbctl lflow-list |grep icmp

如何找到dhcp在哪个host上呢?

用OVN实现OpenStack原理以流表分析

  1. 1, 每个Node(Chassis)上创建br-data2
  2. ovs-vsctl --may-exist add-br br-data2
  3. #ovs-vsctl --may-exist add-port br-data2 eth2
  4. ovs-vsctl set open . external-ids:ovn-bridge-mappings=physnet1:br-data,physnet2:br-data2
  5. 2, 东西向的两个Subnet(Switch)和vRouter(Logical Router)
  6. ovn-nbctl ls-add sw0
  7. ovn-nbctl lsp-add sw0 sw0-port1
  8. ovn-nbctl lsp-set-addresses sw0-port1 "00:00:01:00:00:03 10.0.0.3"
  9. ovn-nbctl ls-add sw1
  10. ovn-nbctl lsp-add sw1 sw1-port1
  11. ovn-nbctl lsp-set-addresses sw1-port1 "00:00:02:00:00:03 20.0.0.3"
  12. ovn-nbctl lr-add lr0
  13. # Connect sw0 to lr0
  14. ovn-nbctl lrp-add lr0 lr0-sw0 00:00:00:00:ff:01 10.0.0.1/24
  15. ovn-nbctl lsp-add sw0 sw0-lr0
  16. ovn-nbctl lsp-set-type sw0-lr0 router
  17. ovn-nbctl lsp-set-addresses sw0-lr0 router
  18. ovn-nbctl lsp-set-options sw0-lr0 router-port=lr0-sw0
  19. # Connect sw1 to lr0
  20. ovn-nbctl lrp-add lr0 lr0-sw1 00:00:00:00:ff:02 20.0.0.1/24
  21. ovn-nbctl lsp-add sw1 sw1-lr0
  22. ovn-nbctl lsp-set-type sw1-lr0 router
  23. ovn-nbctl lsp-set-addresses sw1-lr0 router
  24. ovn-nbctl lsp-set-options sw1-lr0 router-port=lr0-sw1
  25. ovn-nbctl show
  26. 3, 南北向流量,br-ex(public),并在vRouter(lr0)中设置外部IP(即将lr0public关联)
  27. 这里network_namephysnet2, localnet port定义在switch上意味着运行在Gateway Chassis (Centralized L3)
  28. 上的ovn-controller将在br-intbr-data2之前创建patch port
  29. ovn-nbctl ls-add public
  30. # Create a localnet port
  31. ovn-nbctl lsp-add public ln-public
  32. ovn-nbctl lsp-set-type ln-public localnet
  33. ovn-nbctl lsp-set-addresses ln-public unknown
  34. ovn-nbctl lsp-set-options ln-public network_name=physnet2
  35. # 这个vRouter中的外部接口(lr0-public)也应该部署在L3上,它在将流量经patch port转给br-ex前需SNAT
  36. # 并且当有人要访问外部地址172.168.0.200时,Gateway Chassis也需响应ARP reply
  37. ovn-nbctl lrp-add lr0 lr0-public 00:00:20:20:12:13 172.168.0.200/24
  38. ovn-nbctl lsp-add public public-lr0
  39. ovn-nbctl lsp-set-type public-lr0 router
  40. ovn-nbctl lsp-set-addresses public-lr0 router
  41. ovn-nbctl lsp-set-options public-lr0 router-port=lr0-public
  42. 有两个方法schedule Gateway router port(lr0-public):
  43. a, non_HA, 只有single Gateway Chassis, eg: scheduleovn-controller
  44. ovn-nbctl set logical_router_port lr0-public options:redirect-chassis=juju-e28e17-ovn2-8.cloud.sts
  45. # ovn-sbctl show |grep lr0-public
  46. Port_Binding cr-lr0-public
  47. b, HA, 有多个Gateway Chassis, 一个挂了,另外高优先级的能起来,类似于VRRP
  48. ovn-nbctl lrp-set-gateway-chassis lr0-public juju-e28e17-ovn2-8.cloud.sts 20
  49. ovn-nbctl lrp-set-gateway-chassis lr0-public controller-1 15
  50. ovn-nbctl lrp-set-gateway-chassis lr0-public controller-2 10
  51. #ovn-nbctl lrp-del-gateway-chassis lr0-public controller-1
  52. "ovn-nbctl list gateway_chassis"能看到有多个Gateway Chassis
  53. "ovn-nbctl list logical_router_port lr0-public"也能看到多个Gateway Chassis
  54. 对于这个HA模式,OVN使用BFD(Bidirectional Forwarding Detection)协议,它配置BFDtunnel ports上,
  55. 当一个Gateway Chassis上的Distributed gateway port挂了的话,其他的Gateway Chassis上都能检测到
  56. 它的typechassisredirect
  57. # ovn-sbctl list Port_Binding cr-lr0-public |grep type
  58. type : chassisredirect
  59. 4, 南北向出虚机流量的流表如下:
  60. a, vrouter上的内部网关(lr0-sw0)与外部网关(lr0-public)将有下列流表(逻辑流):
  61. table=0 (lr_in_admission ), priority=50 , match=(eth.dst == 00:00:00:00:ff:01 && inport == "lr0-sw0"), action=(next;)
  62. table=7 (lr_in_ip_routing ), priority=49 , match=(ip4.dst == 172.168.0.0/24), action=(ip.ttl--; reg0 = ip4.dst; reg1 = 172.168.0.200; eth.src = 00:00:20:20:12:13; outport = "lr0-public"; flags.loopback = 1; next;)
  63. table=9 (lr_in_gw_redirect ), priority=50 , match=(outport == "lr0-public"), action=(outport = "cr-lr0-public"; next;)
  64. b, 假如 cr-lr0-publicschedulecontroller-0上,包应该走tunnel port
  65. table=32, priority=100,reg15=0x4,metadata=0x3 actions=load:0x3->NXM_NX_TUN_ID[0..23],set_field:0x4->tun_metadata0,move:NXM_NX_REG14[0..14]->NXM_NX_TUN_METADATA0[16..30],output:ovn-cont-0
  66. c, controller-0 chassis收到tunnel port的包,并把它转到lr0
  67. table=0, priority=100,in_port="ovn-comp-0" actions=move:NXM_NX_TUN_ID[0..23]->OXM_OF_METADATA[0..23],move:NXM_NX_TUN_METADATA0[16..30]->NXM_NX_REG14[0..14],move:NXM_NX_TUN_METADATA0[0..15]->NXM_NX_REG15[0..15],resubmit(,33)
  68. d, 接着是SNAT
  69. table=1 (lr_out_snat ), priority=25 , match=(ip && ip4.src == 10.0.0.0/24 && outport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat(172.168.0.200);)
  70. e, 发到vRouter的外网网关lr0-public port
  71. table=3 (lr_out_delivery ), priority=100 , match=(outport == "lr0-public"), action=(output;)
  72. f, 包将localnet port(patch port)进provider bridge(br-ex)并最终到达目的地
  73. 5, 南北向进虚机流量的流表如下:
  74. controller-0 chassis上:
  75. a, 物理网关收到包,走br-ex,再经local net port(patch port)进vRouter(lr0)
  76. table=0,priority=100,in_port="patch-br-int-to",dl_vlan=0 actions=strip_vlan,load:0x1->NXM_NX_REG13[],load:0x7->NXM_NX_REG11[],load:0x8->NXM_NX_REG12[],load:0x4->OXM_OF_METADATA[],load:0x2->NXM_NX_REG14[],resubmit(,8)
  77. b, DNAT(UnSNAT): 172.168.0.200 -> 10.0.0.3
  78. table=0 (lr_in_admission ), priority=50 , match=(eth.dst == 00:00:20:20:12:13 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(next;)
  79. table=3 (lr_in_unsnat ), priority=100 , match=(ip && ip4.dst == 172.168.0.200 && inport == "lr0-public" && is_chassis_resident("cr-lr0-public")), action=(ct_snat;)
  80. c, 10.0.0.3属于Switch sw0, vRouter(lr0)需将流量导到lr0-sw0
  81. table=7 (lr_in_ip_routing ), priority=49 , match=(ip4.dst == 10.0.0.0/24), action=(ip.ttl--; reg0 = ip4.dst; reg1 = 10.0.0.1; eth.src = 00:00:00:00:ff:01; outport = "lr0-sw0"; flags.loopback = 1; next;)
  82. e, The ingress pipeline of sw0 is run and the packet is sent to compute-0 via the tunnel port because OVN knows that sw0-port1 resides on compute-0.
  83. compute-0 Chassic上:
  84. f, compute-0 receives the traffic on the tunnel port and sends the traffic to the egress pipeline of logical switch sw0, In the egress pipeline, the packet is delivered to sw0-port1.

从ovnmeta ns中ping同台节点虚机的流表

  1. 例如tapbdbe0b57-eb/fe:16:3e:ec:f5:9a对应的是虚机,而tape2ed122c-90ovnmeta ns里的tap
  2. ip netns exec ovnmeta-e2ed122c-9442-4116-9793-5c2323f6de29 ip addr show
  3. tape2ed122c-91@if12 -> tape2ed122c-90@if2
  4. # ovs-vsctl -- --columns=name,ofport list Interface tape2ed122c-90
  5. name : tape2ed122c-90
  6. ofport : 3
  7. ovs-vsctl -- --columns=name,ofport list Interface tapbdbe0b57-eb
  8. name : tapbdbe0b57-eb
  9. ofport : 2
  10. # ovs-ofctl -O OpenFlow13 dump-flows br-int |grep 'in_port=3'
  11. cookie=0x3384108f, duration=38141.438s, table=0, n_packets=0, n_bytes=0, priority=100,in_port=3 actions=set_field:0x9->reg13,set_field:0x1->reg11,set_field:0x6->reg12,set_field:0x3->metadata,set_field:0x1->reg14,resubmit(,8)
  12. Flow: in_port=3,vlan_tci=0x0000,dl_src=fe:16:3e:ec:f5:9a,dl_dst=12:d6:31:fe:59:a8,dl_type=0x0000
  13. bridge("br-int")
  14. ----------------
  15. - in_port=3, priority 100, cookie 0x3384108f
  16. set_field:0x9->reg13
  17. set_field:0x1->reg11
  18. set_field:0x6->reg12
  19. set_field:0x3->metadata
  20. set_field:0x1->reg14
  21. resubmit(,8)
  22. - reg14=0x1,metadata=0x3, priority 50, cookie 0x3c9c4889
  23. resubmit(,9)
  24. - metadata=0x3, priority 0, cookie 0x6475678d
  25. resubmit(,10)
  26. - metadata=0x3, priority 0, cookie 0x9593deb1
  27. resubmit(,11)
  28. - metadata=0x3, priority 0, cookie 0x4db625a0
  29. resubmit(,12)
  30. - metadata=0x3, priority 0, cookie 0x395e746e
  31. resubmit(,13)
  32. - metadata=0x3, priority 0, cookie 0x57defbe5
  33. resubmit(,14)
  34. - metadata=0x3, priority 0, cookie 0x32c7898e
  35. resubmit(,15)
  36. - metadata=0x3, priority 0, cookie 0xf06ee7a1
  37. resubmit(,16)
  38. - metadata=0x3, priority 0, cookie 0x351d116c
  39. resubmit(,17)
  40. - metadata=0x3, priority 0, cookie 0x3f2d1d9e
  41. resubmit(,18)
  42. - metadata=0x3, priority 0, cookie 0x42601c1a
  43. resubmit(,19)
  44. - metadata=0x3, priority 0, cookie 0x2b6f92a0
  45. resubmit(,20)
  46. - metadata=0x3, priority 0, cookie 0xabefbd0d
  47. resubmit(,21)
  48. - metadata=0x3, priority 0, cookie 0x4854b450
  49. resubmit(,22)
  50. - metadata=0x3, priority 0, cookie 0xf346b65d
  51. resubmit(,23)
  52. - metadata=0x3, priority 0, cookie 0xb2d4f8c9
  53. resubmit(,24)
  54. - metadata=0x3, priority 0, cookie 0x49d3f6be
  55. resubmit(,25)
  56. - metadata=0x3, priority 0, cookie 0x5bd4e34b
  57. resubmit(,26)
  58. - metadata=0x3, priority 0, cookie 0x8e0cea9
  59. resubmit(,27)
  60. - No match.
  61. drop
  62. Final flow: reg11=0x1,reg12=0x6,reg13=0x9,reg14=0x1,metadata=0x3,in_port=3,vlan_tci=0x0000,dl_src=fe:16:3e:ec:f5:9a,dl_dst=12:d6:31:fe:59:a8,dl_type=0x0000
  63. Megaflow: recirc_id=0,ct_state=-new-est-rel-rpl-inv-trk,ct_label=0/0x1,eth,in_port=3,dl_src=fe:16:3e:ec:f5:9a,dl_dst=12:d6:31:fe:59:a8,dl_type=0x0000
  64. Datapath actions: drop

从ovnmeta ns中ping同台节点虚机的逻辑流表

https://blog.csdn.net/zhengmx100/article/details/78140948
流表是单个机器的(流表由table组成,table包含flow,flow又有priority, match, action),逻辑流表是多台机器编排的(逻辑流表下发到每台机器的ovn-controller,而ovn-controller知道根据当前的物理环境即本地端口如何到达其他机器而转换成流表)。
例如创建一个logical switch sw0, 再创建两个port。sw0将有两个pipeline (ingress pipeline与egress pipline), 两个port port在相同主机与不同相同主机所走的pipeline不一样,如下:
在这里插入图片描述
在这里插入图片描述
逻辑流表就是类似于neutron的port等这些高层概念,它的port只是网络中的一个元素并不限定在某台节点上。可以使用”ovn-sbctl lflow-list”命令来查看完整的logical flow。理解logical flow最好的方式就是使用ovn-trace命令。ovn-trace能够让你看到OVN对一个包是怎么处理的。

  1. ovn-trace DATAPATH MICROFLOW
  2. 例如,还是用ovnmeta ns(192.168.21.2/12:d6:31:fe:59:a8)中来ping VM(192.168.21.18/fe:16:3e:ec:f5:9a)
  3. switch eb7511fa-6746-4981-9ddc-6e1c73e9a7ee (neutron-b3187a23-b54e-4cc0-bc6d-4caabdb02b0c) (aka private)
  4. port bdbe0b57-eb1e-41d5-a06d-b2fd3050fa4d
  5. addresses: ["fa:16:3e:ec:f5:9a 192.168.21.18"]
  6. port a6194e3b-6de5-4334-beb7-a898760488d0
  7. type: localport
  8. addresses: ["fa:16:3e:de:68:05 192.168.21.2"]
  9. ...
  10. # ovn-trace --summary private 'inport == "a6194e3b-6de5-4334-beb7-a898760488d0" && eth.src == 12:d6:31:fe:59:a8 && eth.dst == fe:16:3e:ec:f5:9a'
  11. # reg14=0x1,vlan_tci=0x0000,dl_src=12:d6:31:fe:59:a8,dl_dst=fe:16:3e:ec:f5:9a,dl_type=0x0000
  12. ingress(dp="private", inport="a6194e") {
  13. next;
  14. };
  15. root@juju-e28e17-ovn2-10:/home/ubuntu# ovn-trace --detail private 'inport == "a6194e3b-6de5-4334-beb7-a898760488d0" && eth.src == 12:d6:31:fe:59:a8 && eth.dst == fe:16:3e:ec:f5:9a'
  16. # reg14=0x1,vlan_tci=0x0000,dl_src=12:d6:31:fe:59:a8,dl_dst=fe:16:3e:ec:f5:9a,dl_type=0x0000
  17. ingress(dp="private", inport="a6194e")
  18. --------------------------------------
  19. 0. ls_in_port_sec_l2 (ovn-northd.c:4514): inport == "a6194e", priority 50, uuid 3c9c4889
  20. next;
  21. 19. ls_in_l2_lkup: no match (implicit drop)

Possible bugs

[1] https://bugs.launchpad.net/charm-ovn-chassis/+bug/1907686
[2] https://bugs.launchpad.net/charm-neutron-api/+bug/1921986

Reference

[1] https://networkop.co.uk/blog/2016/12/10/ovn-part2/
[2] https://numans.blog/2018/11/30/how-to-create-an-open-virtual-network-distributed-gateway-router/

发表评论

表情:
评论列表 (有 0 条评论,302人围观)

还没有评论,来说两句吧...

相关阅读

    相关 debug glance(by quqi99)

    作者:张华 发表于:2021-05-19 版权声明:可以任意转载,转载时请务必以超链接形式标明文章原始出处和作者信息及本版权声明 glance与nova都使用rbd的话使