CentOS 7上搭建安全、容灾、高可用的etcd集群

桃扇骨 2022-06-06 20:38 218阅读 0赞

【编者的话】etcd 是 CoreOS 团队发起的开源项目，基于 Go 语言实现，做为一个分布式键值对存储，通过分布式锁，leader选举和写屏障(write barriers)来实现可靠的分布式协作。

本文目标是部署一个基于TLS（Self-signed certificates）的安全、快速灾难恢复（Disaster Recovery, SNAPSHOT）的高可用（High Availability）的etcd集群。

### 准备工作 ###

版本信息：

OS: CentOS Linux release 7.3.1611 (Core) 
    etcd Version: 3.2.4
    Git SHA: c31bec0
    Go Version: go1.8.3
    Go OS/Arch: linux/amd64

#### 机器配置信息 ####

CoreOS官方推荐集群规模5个为宜，为了简化本文仅以3个节点为例：

NAME       ADDRESS             HOSTNAME                    CONFIGURATION
    infra0  192.168.16.227  bjo-ep-kub-01.dev.fwmrm.net  8cpus, 16GB内存, 500GB磁盘
    infra1  192.168.16.228  bjo-ep-kub-02.dev.fwmrm.net  8cpus, 16GB内存, 500GB磁盘
    infra2  192.168.16.229  bjo-ep-kub-03.dev.fwmrm.net  8cpus, 16GB内存, 500GB磁盘

官方建议配置

硬件            通常场景                    重负载
    CPU           2-4 cores                 8-18 cores 
    Memory        8GB                       16GB-64GB
    Disk          50 sequential IOPS        500 sequential IOPS
    Network       1GbE                      10GbE

注：重负载情况以CPU为例，每秒处理数以千计的client端请求。AWS、GCE推荐配置请参考：[Example hardware configurations on AWS and GCE][]

### 搭建etcd集群 ###

搭建etcd集群有3种方式，分别为Static, etcd Discovery, DNS Discovery。Discovery请参见官网[https://coreos.com/etcd/docs/l … .html][https_coreos.com_etcd_docs_l _ .html]，在此不再敖述。本文仅以Static方式展示一次集群搭建过程。  
每个node的etcd配置分别如下：

$ /export/etcd/etcd --name infra0 --initial-advertise-peer-urls http://192.168.16.227:2380 \
    --listen-peer-urls http://192.168.16.227:2380 \
    --listen-client-urls http://192.168.16.227:2379,http://127.0.0.1:2379 \
    --advertise-client-urls http://192.168.16.227:2379 \
    --initial-cluster-token etcd-cluster-1 \
    --initial-cluster infra0=http://192.168.16.227:2380,infra1=http://192.168.16.228:2380,infra2=http://192.168.16.229:2380 \
    --initial-cluster-state new

$ /export/etcd/etcd --name infra1 --initial-advertise-peer-urls http://192.168.16.228:2380 \
    --listen-peer-urls http://192.168.16.228:2380 \
    --listen-client-urls http://192.168.16.228:2379,http://127.0.0.1:2379 \
    --advertise-client-urls http://192.168.16.228:2379 \
    --initial-cluster-token etcd-cluster-1 \
    --initial-cluster infra0=http://192.168.16.227:2380,infra1=http://192.168.16.228:2380,infra2=http://192.168.16.229:2380 \
    --initial-cluster-state new

$ /export/etcd/etcd --name infra2 --initial-advertise-peer-urls http://192.168.16.229:2380 \
    --listen-peer-urls http://192.168.16.229:2380 \
    --listen-client-urls http://192.168.16.229:2379,http://127.0.0.1:2379 \
    --advertise-client-urls http://192.168.16.229:2379 \
    --initial-cluster-token etcd-cluster-1 \
    --initial-cluster infra0=http://192.168.16.227:2380,infra1=http://192.168.16.228:2380,infra2=http://192.168.16.229:2380 \
    --initial-cluster-state new

### TLS ###

etcd支持通过TLS加密通信，TLS channels可被用于集群peer间通信加密，以及client端traffic加密。Self-signed certificates与Automatic certificates两种安全认证形式，其中Self-signed certificates:自签名证书既可以加密traffic也可以授权其连接。本文以Self-signed certificates为例，使用Cloudflare的cfssl很容易生成集群所需证书。  
首先，安装go以及设置环境变量GOPATH

$ cd /export
    $ wget https://storage.googleapis.com/golang/go1.8.3.linux-amd64.tar.gz
    $ tar -xzf go1.8.3.linux-amd64.tar.gz
    
    $ sudo vim ~/.profile
    $ export GOPATH=/export/go_path
    $ export GOROOT=/export/go/
    $ export CFSSL=/export/go_path/
    $ export PATH=$PATH:$GOROOT/bin:$CFSSL/bin
    
    $ source ~/.profile

下载并build CFSSL工具, 安装路径为$GOPATH/bin/cfssl, eg. cfssl, cfssljson会被安装到/export/go\_path目录。

$ go get -u github.com/cloudflare/cfssl/cmd/cfssl
    $ go get -u github.com/cloudflare/cfssl/cmd/cfssljson

初始化certificate authority

$ mkdir ~/cfssl
    $ cd ~/cfssl
    $ cfssl print-defaults config > ca-config.json
    $ cfssl print-defaults csr > ca-csr.json

配置CA选项， ca-config.json文件内容如下

{
       
    "signing": {
       
        "default": {
       
            "expiry": "43800h"
        },
        "profiles": {
       
            "server": {
       
                "expiry": "43800h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth"
                ]
            },
            "client": {
       
                "expiry": "43800h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "client auth"
                ]
            },
            "peer": {
       
                "expiry": "43800h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ]
            }
        }
    }

ca-csr.json Certificate Signing Request (CSR)文件内容如下

{
       
    "CN": "My own CA",
    "key": {
       
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
       
            "C": "US",
            "L": "CA",
            "O": "My Company Name",
            "ST": "San Francisco",
            "OU": "Org Unit 1",
            "OU": "Org Unit 2"
        }
    ]
    }

用已定义的选项生成CA：cfssl gencert -initca ca-csr.json | cfssljson -bare ca –

$ cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
    2017/08/02 00:56:03 [INFO] generating a new CA key and certificate from CSR
    2017/08/02 00:56:03 [INFO] generate received request
    2017/08/02 00:56:03 [INFO] received CSR
    2017/08/02 00:56:03 [INFO] generating key: rsa-2048
    2017/08/02 00:56:04 [INFO] encoded CSR
    2017/08/02 00:56:04 [INFO] signed certificate with serial number 81101109133309828380726760425799837279517519090

会在当前目录下生成如下文件

ca-key.pem
    ca.csr
    ca.pem

注：保存好ca-key.pem文件。

生成server端证书：

$ cfssl print-defaults csr > server.json

server.json内容如下：

{
       
    "CN": "server",
    "hosts": [
        "127.0.0.1",
        "192.168.16.227",
        "192.168.16.228",
        "192.168.16.229",
        "bjo-ep-kub-01.dev.fwmrm.net",
        "bjo-ep-kub-02.dev.fwmrm.net",
        "bjo-ep-kub-03.dev.fwmrm.net"
    ],
    "key": {
       
        "algo": "ecdsa",
        "size": 256
    },
    "names": [
        {
       
            "C": "US",
            "L": "CA",
            "ST": "San Francisco"
        }
    ]
    }

接下来生成server端证书以及private key

$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server server.json | cfssljson -bare server

$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server server.json | cfssljson -bare server
    2017/08/02 00:57:12 [INFO] generate received request
    2017/08/02 00:57:12 [INFO] received CSR
    2017/08/02 00:57:12 [INFO] generating key: ecdsa-256
    2017/08/02 00:57:12 [INFO] encoded CSR
    2017/08/02 00:57:12 [INFO] signed certificate with serial number 138149747694684969550285630966539823697635905885

将会生成如下文件：

server-key.pem
    server.csr
    server.pem

生成peer certificate

$ cfssl print-defaults csr > member1.json

替换 CN和hosts值，如下：

{
       
    "CN": "member1",
    "hosts": [
        "127.0.0.1",
        "192.168.16.227",
        "192.168.16.228",
        "192.168.16.229",
        "bjo-ep-kub-01.dev.fwmrm.net",
        "bjo-ep-kub-02.dev.fwmrm.net",
        "bjo-ep-kub-03.dev.fwmrm.net"
    ],
    "key": {
       
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
       
            "C": "US",
            "ST": "CA",
            "L": "San Francisco"
        }
    ]

生成 member1 certificate与private key

$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=peer member1.json | cfssljson -bare member1

$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=peer member1.json | cfssljson -bare member1
    2017/08/02 00:59:12 [INFO] generate received request
    2017/08/02 00:59:12 [INFO] received CSR
    2017/08/02 00:59:12 [INFO] generating key: rsa-2048
    2017/08/02 00:59:13 [INFO] encoded CSR
    2017/08/02 00:59:13 [INFO] signed certificate with serial number 222573666682951886940627822839805508037201209158

得到如下文件：

member1-key.pem
    member1.csr
    member1.pem

在集群其他节点上重复如上步骤。

生成 client certificate

$ cfssl print-defaults csr > client.json

client.json内容如下：

{
       
    "CN": "client",
    "hosts": [
        "127.0.0.1",
        "192.168.16.227",
        "192.168.16.228",
        "192.168.16.229"
    ],
    "key": {
       
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
       
            "C": "US",
            "ST": "CA",
            "L": "San Francisco"
        }
    ]

生成client certificate

$ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=client client.json | cfssljson -bare client

将会得到如下文件

client-key.pem
    client.csr
    client.pem

#### 拷贝节点1生成的证书到全部节点，并将证书全部置于/etc/ssl/etcd/目录, 至此TLS证书全部生成完成。 ####

#### 测试TLS ####

##### 示例1： 客户端到服务器采用HTTPS客户端证书授权 #####

启动etcd服务：

$ /export/etcd/etcd -name infra0 --data-dir infra0 \
    --client-cert-auth --trusted-ca-file=/etc/ssl/etcd/ca.pem --cert-file=/etc/ssl/etcd/server.pem --key-file=/etc/ssl/etcd/server-key.pem \  
    --advertise-client-urls=https://127.0.0.1:2379 --listen-client-urls=https://127.0.0.1:2379

插入数据：

$ curl --cacert /etc/ssl/etcd/ca.pem --cert /etc/ssl/etcd/client.pem --key /etc/ssl/etcd/client-key.pem -L https://127.0.0.1:2379/v2/keys/foo -XPUT -d value=bar -v

读取数据成功：

$ curl --cacert /etc/ssl/etcd/ca.pem --cert /etc/ssl/etcd/client.pem --key /etc/ssl/etcd/client-key.pem -L https://127.0.0.1:2379/v2/keys/foo
    {
       "action":"get","node":{
       "key":"/foo","value":"bar","modifiedIndex":12,"createdIndex":12

####  示例2：Using self-signed certificates both encrypts traffic and authenticates its connections. ####

各节点的etcd配置分别如下

$ /export/etcd/etcd \
    --name infra0 \
    --initial-advertise-peer-urls https://192.168.16.227:2380 \
    --listen-peer-urls https://192.168.16.227:2380 \
    --listen-client-urls https://192.168.16.227:2379,https://127.0.0.1:2379 \
    --advertise-client-urls https://192.168.16.227:2379 \
    --initial-cluster-token etcd-cluster-1 \
    --initial-cluster infra0=https://192.168.16.227:2380,infra1=https://192.168.16.228:2380,infra2=https://192.168.16.229:2380 \
    --initial-cluster-state new \
    --client-cert-auth --trusted-ca-file=/etc/ssl/etcd/ca.pem \
    --cert-file=/etc/ssl/etcd/server.pem --key-file=/etc/ssl/etcd/server-key.pem \
    --peer-client-cert-auth --peer-trusted-ca-file=/etc/ssl/etcd/ca.pem \
    --peer-cert-file=/etc/ssl/etcd/member1.pem --peer-key-file=/etc/ssl/etcd/member1-key.pem

$ /export/etcd/etcd \
    --name infra1 \
    --initial-advertise-peer-urls https://192.168.16.228:2380 \
    --listen-peer-urls https://192.168.16.228:2380 \
    --listen-client-urls https://192.168.16.228:2379,https://127.0.0.1:2379 \
    --advertise-client-urls https://192.168.16.228:2379 \
    --initial-cluster-token etcd-cluster-1 \
    --initial-cluster infra0=https://192.168.16.227:2380,infra1=https://192.168.16.228:2380,infra2=https://192.168.16.229:2380 \
    --initial-cluster-state new \
    --client-cert-auth --trusted-ca-file=/etc/ssl/etcd/ca.pem \
    --cert-file=/etc/ssl/etcd/server.pem --key-file=/etc/ssl/etcd/server-key.pem \
    --peer-client-cert-auth --peer-trusted-ca-file=/etc/ssl/etcd/ca.pem \
    --peer-cert-file=/etc/ssl/etcd/member2.pem --peer-key-file=/etc/ssl/etcd/member2-key.pem

$ /export/etcd/etcd \
    --name infra2 \
    --initial-advertise-peer-urls https://192.168.16.229:2380 \
    --listen-peer-urls https://192.168.16.229:2380 \
    --listen-client-urls https://192.168.16.229:2379,https://127.0.0.1:2379 \
    --advertise-client-urls https://192.168.16.229:2379 \
    --initial-cluster-token etcd-cluster-1 \
    --initial-cluster infra0=https://192.168.16.227:2380,infra1=https://192.168.16.228:2380,infra2=https://192.168.16.229:2380 \
    --initial-cluster-state new \
    --client-cert-auth --trusted-ca-file=/etc/ssl/etcd/ca.pem \
    --cert-file=/etc/ssl/etcd/server.pem --key-file=/etc/ssl/etcd/server-key.pem \
    --peer-client-cert-auth --peer-trusted-ca-file=/etc/ssl/etcd/ca.pem \
    --peer-cert-file=/etc/ssl/etcd/member3.pem --peer-key-file=/etc/ssl/etcd/member3-key.pem

准备测试数据：

$ curl --cacert /etc/ssl/etcd/ca.pem --cert /etc/ssl/etcd/client.pem --key /etc/ssl/etcd/client-key.pem -L https://127.0.0.1:2379/v2/keys/fristname -XPUT -d value=Xia -v
    
    $ ETCDCTL_API=3 /export/etcd/etcdctl --cacert /etc/ssl/etcd/ca.pem --cert /etc/ssl/etcd/client.pem --key /etc/ssl/etcd/client-key.pem --endpoints=https://192.168.16.227:2379,https://192.168.16.228:2379,https://192.168.16.229:2379 put lasttname 'Zhang'

验证测试结果：

$ curl --cacert /etc/ssl/etcd/ca.pem --cert /etc/ssl/etcd/client.pem --key /etc/ssl/etcd/client-key.pem -L https://127.0.0.1:2379/v2/keys/
    {
       "action":"get","node":{
       "dir":true,"nodes":[{
       "key":"/foo","value":"bar","modifiedIndex":19,"createdIndex":19},{
       "key":"/fristname","value":"Xia","modifiedIndex":20,"createdIndex":20},{
       "key":"/lasttname","value":"Zhang","modifiedIndex":21,"createdIndex":21}]

#### etcd Troubleshooting ####

##### etcd failure主要分为如下5种情况： #####

*  少数followers failure
 *  Leader failure
 *  多数failure
 *  Network partition
 *  启动时失败

接下来主要对上面情况第三种进行处理，也就是平时常说的Disaster Recovery

### 灾备恢复（Disaster Recovery） ###

以etcd v3 provides snapshot 方式为例说明etcd一次灾难恢复过程。  
首先，etcd正常工作时利用etcdctl snapshot save命令或拷贝etcd目录中的member/snap/db文件，以前者为例：

$ ETCDCTL_API=3 etcdctl --endpoints $ENDPOINT snapshot save snapshot.db}}
    如果enable TLS，需要如下命令：
    {
       {
       {
       $ ETCDCTL_API=3 /export/etcd/etcdctl --endpoints=https://192.168.16.227:2379,https://192.168.16.228:2379,https://192.168.16.228:2379 snapshot save snapshot.db --cacert=/etc/ssl/etcd/ca.pem --cert=/etc/ssl/etcd/client.pem --key=/etc/ssl/etcd/client-key.pem
    
    Snapshot saved at snapshot.db

将生成snapshot拷贝到集群其他2个节点上，所有节点灾备的恢复都用同一个snapshot。  
插入部分数据用于测试灾备：

测试数据已插入成功：

$ ETCDCTL_API=3 /export/etcd/etcdctl --cacert /etc/ssl/etcd/ca.pem --cert /etc/ssl/etcd/client.pem --key /etc/ssl/etcd/client-key.pem --endpoints=https://192.168.16.227:2379,https://192.168.16.228:2379,https://192.168.16.229:2379  get  firstname
    
    $ curl --cacert /etc/ssl/etcd/ca.pem --cert /etc/ssl/etcd/client.pem --key /etc/ssl/etcd/client-key.pem -L https://127.0.0.1:2379/v2/keys/
    {
       "action":"get","node":{
       "dir":true,"nodes":[{
       "key":"/foo","value":"bar","modifiedIndex":19,"createdIndex":19},{
       "key":"/fristname","value":"Xia","modifiedIndex":20,"createdIndex":20},{
       "key":"/lasttname","value":"Zhang","modifiedIndex":21,"createdIndex":21}]

停止3个机器的etcd服务，并删除全部节点上etcd数据目录 。  
恢复数据，以TLS enable为例，分别在3个节点执行如下命令进行恢复：

$ ETCDCTL_API=3 /export/etcd/etcdctl snapshot restore snapshot.db \
    --name infra0 \
    --initial-cluster infra0=https://192.168.16.227:2380,infra1=https://192.168.16.228:2380,infra2=https://192.168.16.229:2380 \
    --initial-cluster-token etcd-cluster-1 \
    --initial-advertise-peer-urls https://192.168.16.227:2380 \
    --cacert /etc/ssl/etcd/ca.pem \
    --cert /etc/ssl/etcd/client.pem \
    --key /etc/ssl/etcd/client-key.pem

$ ETCDCTL_API=3 /export/etcd/etcdctl snapshot restore snapshot.db \
    --name infra1 \
    --initial-cluster infra0=https://192.168.16.227:2380,infra1=https://192.168.16.228:2380,infra2=https://192.168.16.229:2380 \
    --initial-cluster-token etcd-cluster-1 \
    --initial-advertise-peer-urls https://192.168.16.228:2380 \
    --cacert /etc/ssl/etcd/ca.pem \
    --cert /etc/ssl/etcd/client.pem \
    --key /etc/ssl/etcd/client-key.pem

$ ETCDCTL_API=3 /export/etcd/etcdctl snapshot restore snapshot.db \
    --name infra2 \
    --initial-cluster infra0=https://192.168.16.227:2380,infra1=https://192.168.16.228:2380,infra2=https://192.168.16.229:2380 \
    --initial-cluster-token etcd-cluster-1 \
    --initial-advertise-peer-urls https://192.168.16.229:2380 \
    --cacert /etc/ssl/etcd/ca.pem \
    --cert /etc/ssl/etcd/client.pem \
    --key /etc/ssl/etcd/client-key.pem

恢复数据log示例：

$ ETCDCTL_API=3 /export/etcd/etcdctl snapshot restore snapshot.db   --name infra0   --initial-cluster infra0=https://192.168.16.227:2380,infra1=https://192.168.16.228:2380,infra2=https://192.168.16.229:2380   --initial-cluster-token etcd-cluster-1   --initial-advertise-peer-urls https://192.168.16.227:2380   --cacert /etc/ssl/etcd/ca.pem   --cert /etc/ssl/etcd/client.pem   --key /etc/ssl/etcd/client-key.pem
    2017-08-06 04:09:12.853510 I | etcdserver/membership: added member 3e5097be4ea17ebe [https://192.168.16.229:2380] to cluster cabc8098aa3afc98
    2017-08-06 04:09:12.853567 I | etcdserver/membership: added member 67d47e92a1704b1a [https://192.168.16.227:2380] to cluster cabc8098aa3afc98
    2017-08-06 04:09:12.853583 I | etcdserver/membership: added member b4725a5341abf1a0 [https://192.168.16.228:2380] to cluster cabc8098aa3afc98

接下来，在3个节点上分别执行：

验证灾备恢复效果，原集群数据是否保存：

$ ETCDCTL_API=3 /export/etcd/etcdctl --cacert /etc/ssl/etcd/ca.pem --cert /etc/ssl/etcd/client.pem --key /etc/ssl/etcd/client-key.pem --endpoints=https://192.168.16.227:2379,https://192.168.16.228:2379,https://192.168.16.229:2379 get lasttname
    lasttname
    Zhang
    
    $ ETCDCTL_API=3 /export/etcd/etcdctl --cacert /etc/ssl/etcd/ca.pem --cert /etc/ssl/etcd/client.pem --key /etc/ssl/etcd/client-key.pem --endpoints=https://192.168.16.227:2379,https://192.168.16.228:2379,https://192.168.16.229:2379 get firstname
    firstname
    Xia

从上面结果可以看出，灾备恢复成功。

#### etcd系统限制 ####

*  请求大小限制：当前支持 RPC requests 1MB 数据，未来会有所增加或可配置
 *  存储大小限制：默认 2GB存储，可配置 –quota-backend-bytes扩展到8GB

### 监控 ###

etcd提供基于Prometheus + builtin Grafana的etcd Metrics监控方案和监控项，具体请参见  
etcd Metrics: [https://coreos.com/etcd/docs/latest/metrics.html][https_coreos.com_etcd_docs_latest_metrics.html]

获取监控项举例：

$  curl --cacert /etc/ssl/etcd/ca.pem --cert /etc/ssl/etcd/client.pem --key /etc/ssl/etcd/client-key.pem -L https://127.0.0.1:2379/metrics
    
    etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket{
       le="1"} 0
    etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket{
       le="2"} 0
    etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket{
       le="4"} 0
    etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket{
       le="8"} 0
    
    ... ...
    
    process_start_time_seconds 1.50390583624e+09
    process_virtual_memory_bytes 1.0787151872e+10

Prometheus + builtin Grafana: [https://coreos.com/etcd/docs/l … .html][https_coreos.com_etcd_docs_l _ .html 1]

欢迎转载，请注明作者出处：张夏，FreeWheel Lead Engineer，[Kubernetes中文社区][Kubernetes]

原文：[CentOS 7上搭建安全、容灾、高可用的etcd集群][CentOS 7_etcd]

[Example hardware configurations on AWS and GCE]: https://coreos.com/etcd/docs/latest/op-guide/hardware.html
[https_coreos.com_etcd_docs_l _ .html]: https://coreos.com/etcd/docs/latest/op-guide/clustering.html
[https_coreos.com_etcd_docs_latest_metrics.html]: https://coreos.com/etcd/docs/latest/metrics.html
[https_coreos.com_etcd_docs_l _ .html 1]: https://coreos.com/etcd/docs/latest/op-guide/monitoring.html
[Kubernetes]: https://www.kubernetes.org.cn
[CentOS 7_etcd]: https://www.kubernetes.org.cn/2626.html