redis cluster:cluster_state:fail/(error) CLUSTERDOWN The cluster is down

拼搏现实的明天。 2023-01-23 13:49 10阅读 0赞

场景：

pt环境下研发反馈说redis集群down 掉了，报(error) CLUSTERDOWN The cluster is down

由于是在k8s 里运行redis-cluster，随便登录上其中一个节点去查询，发现在82这个节点上cluster\_state状态是fail，那么问题就定位到82这台机器上，网上也搜索了一下问题原因好像是slots不连续造成的，那试着网上的办法去解决：

1.登录出现问题的那个节点，运行cluster info

10.42.230.82:6379> cluster info 
    cluster_state:fail
    cluster_slots_assigned:16380
    cluster_slots_ok:16380
    cluster_slots_pfail:0
    cluster_slots_fail:0
    cluster_known_nodes:10
    cluster_size:5
    cluster_current_epoch:51
    cluster_my_epoch:11
    cluster_stats_messages_ping_sent:5190185
    cluster_stats_messages_pong_sent:5617414
    cluster_stats_messages_meet_sent:389904
    cluster_stats_messages_fail_sent:52
    cluster_stats_messages_auth-ack_sent:1
    cluster_stats_messages_sent:11197556
    cluster_stats_messages_ping_received:5231452
    cluster_stats_messages_pong_received:5579773
    cluster_stats_messages_meet_received:385962
    cluster_stats_messages_fail_received:6
    cluster_stats_messages_auth-req_received:1
    cluster_stats_messages_received:11197194

执行cluster slots 查看槽点分布情况

10.42.230.82:6379> get nodes
    (error) CLUSTERDOWN The cluster is down
    10.42.230.82:6379> get node
    (error) CLUSTERDOWN The cluster is down
    10.42.230.82:6379> get name
    (error) CLUSTERDOWN The cluster is down
    10.42.230.82:6379> get names
    (error) CLUSTERDOWN The cluster is down
    10.42.230.82:6379> cluster slots
    1) 1) (integer) 3278
       2) (integer) 6553
       3) 1) "10.42.209.18"
          2) (integer) 6379
          3) "eb3160f72b96b22b95963dfdde346b4a10bac834"
       4) 1) "10.42.230.80"
          2) (integer) 6379
          3) "46daeb573cd80ff2a8724351b3e16a986b0ffdaa"
    2) 1) (integer) 9831
       2) (integer) 13106
       3) 1) "10.42.230.79"
          2) (integer) 6379
          3) "3cfe4da28c8e3c951e0ea8f1353c2657f11f56f9"
       4) 1) "10.42.34.111"
          2) (integer) 6379
          3) "34db05f361696fc0d23af96fa2a3a9c22c633f16"
    3) 1) (integer) 1
       2) (integer) 3276
       3) 1) "10.42.209.17"
          2) (integer) 6379
          3) "f9ebe7de3aa3ebe229c6cdef8fc9cd4c52c006e6"
       4) 1) "10.42.230.81"
          2) (integer) 6379
          3) "5a3271993366e0023fd9aa53049f65a120d8990b"
    4) 1) (integer) 6554
       2) (integer) 9829
       3) 1) "10.42.230.82"
          2) (integer) 6379
          3) "668d00003902026832f71d531b7ed7f033f7f117"
       4) 1) "10.42.209.16"
          2) (integer) 6379
          3) "cf044f2bd5d8f22d210ce401a350aa1d719500f3"
    5) 1) (integer) 13108
       2) (integer) 16383
       3) 1) "10.42.217.210"
          2) (integer) 6379
          3) "66318424adc132494c57124c6748c06e2abcccb6"
       4) 1) "10.42.209.15"
          2) (integer) 6379
          3) "49140eb3128da7ed80c8c1355a375d109a59fe2b"

统计一下slots,结果如下：

10.42.209.18  slots: 3278~6553 
    10.40.230.82  slots:6554~9829
    10.42.230.79  slots: 9831~13106  
    10.42.217.210 slots:13108~16383

发现slots，缺少0 3277 9830 13107 4个槽点，下面按照网上说的办法在出问题的机器上手动分配一下：

10.42.230.82:6379> CLUSTER ADDSLOTS 0 3277 9830 13107
    OK

再次查看集群状态，发现状态已经ok，让测试那边也验证了一下一切正常。

10.42.230.82:6379> cluster info
    cluster_state:ok
    cluster_slots_assigned:16384
    cluster_slots_ok:16384
    cluster_slots_pfail:0
    cluster_slots_fail:0
    cluster_known_nodes:10
    cluster_size:5
    cluster_current_epoch:51
    cluster_my_epoch:11
    cluster_stats_messages_ping_sent:5192739
    cluster_stats_messages_pong_sent:5620047
    cluster_stats_messages_meet_sent:390015
    cluster_stats_messages_fail_sent:52
    cluster_stats_messages_auth-ack_sent:1
    cluster_stats_messages_sent:11202854
    cluster_stats_messages_ping_received:5234004
    cluster_stats_messages_pong_received:5582438
    cluster_stats_messages_meet_received:386043
    cluster_stats_messages_fail_received:6
    cluster_stats_messages_auth-req_received:1
    cluster_stats_messages_update_received:13
    cluster_stats_messages_received:11202505

到此问题已经解决了，但是为啥会出现这种情况呢，redis环境也没有异常断电重启之类的操作。