Openshift的网络之六：Router和ipfailover

Bertha 。 2022-05-26 05:56 377阅读 0赞

**一、前言**

在文章[《Kubernetes如何利用iptables对外暴露service》][Kubernetes_iptables_service]中提到了部署在Kubernetes上的服务对外暴露的方法，比如通过host port、cluster service、node external IP、external load balancer、node port等，这都是借助于iptables的功能在三层和四层进行的。

在Openshift中，提供了另外一种基于七层的后端服务暴露机制，那就是[Openshift Router][]。Router是一个基于HAProxy的应用层代理，启动之后监听在POD宿主机机的端口，当请求过来之后，会由HAProxy根据URL或者内容进行后端POD的选则，发送代理请求，再将响应发送到客户端。

为了保证后端服务访问接口的健壮性，需要启动多个router的replica，然后将这些replica组成一个cluster，共享一个VIP，提供active-standby的服务模式，这个模式需要启动Openshift的[ipfailover功能][ipfailover]。

ipfailover管理一个VIP的池，每个VIP都代表着后端一系列POD或者一个service，VIP绑定在任何一个或者多个健康的工作节点上。ipfailover借助keepalived，使用VRRP协议通过IP floating保证VIP永远绑定在健康的节点上，只要后端服务是健康的，VIP就可以提供对外的服务。

载自https://blog.csdn.net/cloudvtech

**二、Openshift Router和IPFailover一起工作**

多个Openshift Router可以组成一个cluster，这个cluster可以为多个VIP提供接入服务。多个Router分布在多个工作节点上，VIP可以绑定在有router运行的任意一个工作节点上，一个工作节点只有一个router但是可以绑定多个VIP。

![70][]

这个方案的处理流程如下：

*  keepalived在所有运行Router的节点之间相互通信，为每个VIP选择一个active master对外advertise这个VIP和MAC
 *  客户端向VIP发送的请求到达之后，由HAProxy进行处理
 *  HAProxy根据URL和请求内容，选择合适的后端POD
 *  通过iptables进行四层数据包NAT然后由OVS SDN将数据包转送到POD所在节点
 *  在POD所在节点经过NAT操作进入POD
 *  POD处理请求并产生返回数据包
 *  数据包经上述步骤的反方向操作返回接入HAProxy
 *  HAProxy将数据返回给客户端

载自https://blog.csdn.net/cloudvtech

**三、在Openshift配置Router和IPFailover**

**3.1 Openshift运行环境**

Master: 200.222.0.72
    node1: 200.222.0.73
    node2: 200.222.0.74
    docker registry: 200.222.243

**3.2 为node打label**

oc label nodes node1.os.com "ipf=yes"
    oc label nodes node2.os.com "ipf=yes”
    
    oc get nodes --selector="ipf=yes"
    NAME           STATUS    AGE
    node1.os.com   Ready     7d
    node2.os.com   Ready     7d

**3.3 建立一个Openshift project并建立一个node port service**

oc login -u system:admin -n ipf-test
    kubectl run source-ip-app --image=gcr.io/google_containers/echoserver:1.4
    kubectl expose deployment source-ip-app --name=nodeport --port=8080 --target-port=8080 --type=NodePort

在Openshift GUI设置这个deployment的replica为2

**3.4 在Openshift GUI设置Router的数目为2**

**3.5 为node port service设置route**

![70 1][]

这个操作将使得HAProxy可以根据后端的POD的变化修改的配置

载自https://blog.csdn.net/cloudvtech

**3.6 启动Openshift IPFailover**

oadm policy add-scc-to-user privileged -z router
    oadm ipfailover ipf-ha-router-primary     --replicas=2     --watch-port=8080     --selector="ipf=yes"     --virtual-ips="200.222.0.185-186"     --service-account=router     --create

**3.7 访问VIP**

修改/etc/hosts加入：

200.222.0.185  [nodeport-ipf-test.router.default.svc.cluster.local][]

或者200.222.0.186  [nodeport-ipf-test.router.default.svc.cluster.local][]

curl [nodeport-ipf-test.router.default.svc.cluster.local/ipf][nodeport-ipf-test.router.default.svc.cluster.local_ipf]

可以看到请求会轮流进入两个POD

**3.8 一些日志**

root@node2 ~]# docker logs -f 942a9a0f0ecc
      - Loading ip_vs module ...
      - Checking if ip_vs module is available ...
    ip_vs                 141092  0
      - Module ip_vs is loaded.
      - check for iptables rule for keepalived multicast (224.0.0.18) ...
      - Generating and writing config to /etc/keepalived/keepalived.conf
      - Starting failover services ...
    Starting Healthcheck child process, pid=112
    Initializing ipvs 2.6
    Starting VRRP child process, pid=113
    Netlink reflector reports IP 10.79.47.144 added
    Netlink reflector reports IP 200.222.0.74 added
    Netlink reflector reports IP 200.222.0.185 added
    Netlink reflector reports IP 172.17.0.1 added
    Netlink reflector reports IP 10.128.0.1 added
    Netlink reflector reports IP 10.79.47.144 added
    Netlink reflector reports IP fe80::6ee6:ea71:3f4d:b641 added
    Netlink reflector reports IP fe80::828e:c0f4:d54e:de2 added
    Netlink reflector reports IP fe80::16e6:7310:9226:467e added
    Netlink reflector reports IP fe80::3c44:b7fd:2c7f:9189 added
    Netlink reflector reports IP 200.222.0.74 added
    Netlink reflector reports IP fe80::5840:b3ff:fe3c:d731 added
    Netlink reflector reports IP fe80::24d2:3eff:fe34:dcec added
    Netlink reflector reports IP 200.222.0.185 added
    Netlink reflector reports IP 172.17.0.1 added
    Registering Kernel netlink reflector
    Netlink reflector reports IP 10.128.0.1 added
    Registering Kernel netlink command channel
    Netlink reflector reports IP fe80::6ee6:ea71:3f4d:b641 added
    Netlink reflector reports IP fe80::828e:c0f4:d54e:de2 added
    Opening file '/etc/keepalived/keepalived.conf'.
    Netlink reflector reports IP fe80::16e6:7310:9226:467e added
    Netlink reflector reports IP fe80::3c44:b7fd:2c7f:9189 added
    Netlink reflector reports IP fe80::5840:b3ff:fe3c:d731 added
    Netlink reflector reports IP fe80::24d2:3eff:fe34:dcec added
    Registering Kernel netlink reflector
    Registering Kernel netlink command channel
    Configuration is using : 8923 Bytes
    Registering gratuitous ARP shared channel
    Opening file '/etc/keepalived/keepalived.conf'.
    Truncating auth_pass to 8 characters
    Truncating auth_pass to 8 characters
    Configuration is using : 71477 Bytes
    Using LinkWatch kernel netlink reflector...
    VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(9,10)]
    Using LinkWatch kernel netlink reflector...
    VRRP_Script(chk_ipf_ha_router_primary) succeeded
    VRRP_Instance(ipf_ha_router_primary_VIP_1) Transition to MASTER STATE
    VRRP_Instance(ipf_ha_router_primary_VIP_1) Received lower prio advert, forcing new election
    VRRP_Instance(ipf_ha_router_primary_VIP_2) Transition to MASTER STATE
    VRRP_Instance(ipf_ha_router_primary_VIP_2) Received higher prio advert
    VRRP_Instance(ipf_ha_router_primary_VIP_2) Entering BACKUP STATE
    VRRP_Instance(ipf_ha_router_primary_VIP_1) Entering MASTER STATE
    VRRP_Instance(ipf_ha_router_primary_VIP_1) setting protocol VIPs.
    VRRP_Instance(ipf_ha_router_primary_VIP_1) Sending gratuitous ARPs on ens160 for 200.222.0.185
    Netlink reflector reports IP 200.222.0.185 added
    VRRP_Instance(ipf_ha_router_primary_VIP_1) Sending gratuitous ARPs on ens160 for 200.222.0.185

载自https://blog.csdn.net/cloudvtech

[Kubernetes_iptables_service]: https://blog.csdn.net/cloudvtech/article/details/79728920
[Openshift Router]: https://docs.openshift.com/container-platform/3.7/install_config/router/index.html
[ipfailover]: https://docs.openshift.org/latest/admin_guide/high_availability.html
[70]: /images/20220526/70719ba258e045f496afc65e15e6aa77.png
[70 1]: /images/20220526/c553d9a7ffe544ef9b5b84b1a2a8e1d6.png
[nodeport-ipf-test.router.default.svc.cluster.local]: http://nodeport-ipf-test.router.default.svc.cluster.local
[nodeport-ipf-test.router.default.svc.cluster.local_ipf]: http://nodeport-ipf-test.router.default.svc.cluster.local/ipf