Elasticsearch集群管理篇

落日映苍穹つ 2021-12-18 15:53 293阅读 0赞

Elasticsearch集群管理篇

Elasticsearch出现问题，最高效的解决方案是第一手资料ES英文官网文档，其次是ES英文论坛、ES github issues，再次是stackoverflow等英文论坛、博客。最后才是：Elasticsearch中文社区、其他相关中文技术博客等。

## 1.配置节点 ##

步骤1.修改文件jvm.options

-Xms8g
    -Xmx8g

步骤2.修改文件elasticsearch.yml

# ======================== ES 参数配置 =========================
    #
    #
    # ------------------------ 集群设定 ----------------------------
    #
    # 集群名称 
     cluster.name: benchmark612
    #
    # ------------------------ 节点设定 ----------------------------
    #
    # 节点名称
     node.name: ${HOSTNAME}
    #
    # 节点角色
     node.master: true
     node.data: false
     node.ingest: false
    #
    # ------------------------ 路径设定 ----------------------------
    #
    # 索引、日志存放路径
     path:
       data: /data/store/es-6.1.2_benchmark612
       logs: /data/logs/es-6.1.2_benchmark612
    #
    # ------------------------ 内存设定 ----------------------------
    #
    #
    # 锁定内存，阻止操作系统管理内存，可以有效的防止内存数据被交换到磁盘空间，
    #   交换过程中磁盘会抖动，会对性能产生较大的影响。因为ES是基于JAVA开发的
    #   可以能过垃圾回收器来单独管理内存，所以关闭操作系统级别的内存管理可以
    #   提升性能
     bootstrap.memory_lock: true
    #
    # ------------------------ 网络设定 ----------------------------
    #
    # 绑定节点上的所有网络接口，用于接收通过任意网卡传输过来的请求
     network.bind_host: 0.0.0.0
    #
    # 绑定一个网络接口(网卡)，用于集群内部节点通信(一般选择吞吐量大的网卡)
     network.publish_host: _eth0:ipv4_
    #
    # HTTP 通信端口
     http.port: 50000
    #
    # TCP 通信端口
     transport.tcp.port: 50100
    #
    # --------------------------------- 集群发现 模块 ----------------------------------
    #
    # 集群初始化连接列表，节点启动后，首先通过连接初始化列表里的地址去发现集群。
     discovery.zen.ping.unicast.hosts: ["20.120.203.74:50100","20.120.203.76:50100","20.120.203.81:50100"]
    #
    # 为了防止集群脑裂，目前的策略是当且仅当节点有超过半数的master候选者存活时(目前是2台，可以完成选举)，集群才会进行master选举
     discovery.zen.minimum_master_nodes: 2
    #
    # ---------------------------------- 其它 -----------------------------------
    #
    # 关闭操作系统内核验证(我的操作系统没有升级，如果不关闭验证则无法启动)
     bootstrap.system_call_filter: false
    #
    # ------------------------ HTTP ----------------------------
    #
    # 是否支持跨域访问资源
     http.cors.enabled: true
    #
    #
    #允许访问资源的类型
     http.cors.allow-origin: "*"
    #
    #
    # 允许HTTP请求的方法类型 
     http.cors.allow-methods: OPTIONS,HEAD,GET,POST,PUT,DELETE
    #
    # 允许HTTP请求头返回类型
     http.cors.allow-headers: X-Requested-With,Content-Type,Content-Length,Authorization,Content-Encoding,Accept-Encoding
    #
    # 支持HTTP访问API 总开关
     http.enabled: true
    #
    #

## 2. 集群操作 ##

停止elasticsearch应用，关闭所有的节点  
方法1：  
POST [http://localhost:9200/\_shutdown][http_localhost_9200_shutdown]  
方法2：  
POST [http://localhost:9200/\_cluster/nodes/\_shutdown][http_localhost_9200_cluster_nodes_shutdown]  
方法3：  
POST [http://localhost:9200/\_cluster/nodes/\_all/\_shutdown][http_localhost_9200_cluster_nodes_all_shutdown]

[http://localhost:9200/\_cluster/settings][http_localhost_9200_cluster_settings]

查看ES是否启动成功，查看集群健康状态：  
[http://localhost:9200/\_cluster/health?pretty][http_localhost_9200_cluster_health_pretty]  
返回结果  
\{  
“cluster\_name” : “kevin-elk”, \#集群名称  
“status” : “green”, \#为 green 则代表健康没问题，如果是 yellow 或者 red 则是集群有问题  
“timed\_out” : false, \#是否有超时  
“number\_of\_nodes” : 3, \#集群中的节点数量  
“number\_of\_data\_nodes” : 3,  
“active\_primary\_shards” : 2234,  
“active\_shards” : 4468,  
“relocating\_shards” : 0,  
“initializing\_shards” : 0,  
“unassigned\_shards” : 0,  
“delayed\_unassigned\_shards” : 0,  
“number\_of\_pending\_tasks” : 0,  
“number\_of\_in\_flight\_fetch” : 0,  
“task\_max\_waiting\_in\_queue\_millis” : 0,  
“active\_shards\_percent\_as\_number” : 100.0 \#集群分片的可用性百分比，如果为0则表示不可用  
\}

查看集群健康状态  
[http://localhost:9200/\_cat/health?v][http_localhost_9200_cat_health_v]

检查集群状态：  
[http://localhost:9200/\_cluster/stats?pretty][http_localhost_9200_cluster_stats_pretty]

## 3. 节点操作 ##

关闭指定节点  
POST [http://localhost:9200/\_cluster/nodes/node-2/\_shutdown][http_localhost_9200_cluster_nodes_node-2_shutdown]  
POST [http://localhost:9200/\_cluster/nodes/nodeId1,nodeId2/\_shutdown][http_localhost_9200_cluster_nodes_nodeId1_nodeId2_shutdown]

默认情况下，关闭命令会延迟1秒(1s)之后执行。可以通过设置 delay 参数 来指定延迟的时间。比如：  
POST [http://localhost:9200/\_cluster/nodes/\_local/\_shutdown?delay=10s][http_localhost_9200_cluster_nodes_local_shutdown_delay_10s]

查看节点状态  
curl [http://localhost:9200/\_nodes/process?pretty][http_localhost_9200_nodes_process_pretty]  
curl [http://localhost:9200/\_nodes/node1/process?pretty][http_localhost_9200_nodes_node1_process_pretty]

查看节点状态 (\*表示ES集群的master主节点)  
[http://localhost:9200/\_cat/nodes?v][http_localhost_9200_cat_nodes_v]

显示master节点的id、ip和节点名。  
GET http://localhost:9200\_cat/master?v

查看每个节点的分片数量以及每个节点的磁盘空间使用情况。  
GET http://localhost:9200\_cat/allocation?v

查看每个数据节点上被fielddata所使用的堆内存大小。  
GET http://localhost:9200\_cat/fielddata?v

节点所运行插件信息。  
GET [http://localhost:9200/\_cat/plugins?v][http_localhost_9200_cat_plugins_v]

查看磁盘利用率：  
[http://localhost:9200/\_cat/allocation?v][http_localhost_9200_cat_allocation_v]

## 4. 分片操作 ##

指定延迟多长时间后才开始分配unassigned的分片（默认1分钟）  
PUT /\_all/\_settings  
\{  
“settings”: \{  
“index.unassigned.node\_left.delayed\_timeout”: “5m”  
\}  
\}

查看所有分片状态  
[http://localhost:9200/\_cat/shards?v][http_localhost_9200_cat_shards_v]  
[http://localhost:9200/\_cat/shards?h=index,shard,prirep,state,unassigned.reason][http_localhost_9200_cat_shards_h_index_shard_prirep_state_unassigned.reason]

查看分片没有被分配的原因，比如通过GET /\_cat/shards?v看到某个索引没有被分配，就可以使用下面的命令来查看没有被分配的原因。  
GET [http://localhost:9200/\_cluster/allocation/explain][http_localhost_9200_cluster_allocation_explain]  
\{  
“index”:“twitter”,  
“shard”:0,  
“primary”:true  
\}

分片未分配的原因主要有：  
1）INDEX\_CREATED：由于创建索引的API导致未分配。  
2）CLUSTER\_RECOVERED ：由于完全集群恢复导致未分配。  
3）INDEX\_REOPENED ：由于打开open或关闭close一个索引导致未分配。  
4）DANGLING\_INDEX\_IMPORTED ：由于导入dangling索引的结果导致未分配。  
5）NEW\_INDEX\_RESTORED ：由于恢复到新索引导致未分配。  
6）EXISTING\_INDEX\_RESTORED ：由于恢复到已关闭的索引导致未分配。  
7）REPLICA\_ADDED：由于显式添加副本分片导致未分配。  
8）ALLOCATION\_FAILED ：由于分片分配失败导致未分配。  
9）NODE\_LEFT ：由于承载该分片的节点离开集群导致未分配。  
10）REINITIALIZED ：由于当分片从开始移动到初始化时导致未分配（例如，使用影子shadow副本分片）。  
11）REROUTE\_CANCELLED ：作为显式取消重新路由命令的结果取消分配。  
12）REALLOCATED\_REPLICA ：确定更好的副本位置被标定使用，导致现有的副本分配被取消，出现未分配。

手动指定主分片（旧数据）  
POST [http://localhost:9200/\_cluster/reroute][http_localhost_9200_cluster_reroute]  
\{  
“commands” : \[ \{  
“allocate\_stale\_primary” : \{  
“index” : “idx\_com\_jobs\_v1”,  
“shard” : 1,  
“node” : “node191”,  
“accept\_data\_loss” : true  
\}  
\}\]  
\}

手动指定主分片（空数据）  
POST [http://localhost:9200/\_cluster/reroute][http_localhost_9200_cluster_reroute]  
\{  
“commands” : \[ \{  
“allocate\_empty\_primary” : \{  
“index” : “idx\_com\_jobs\_v1”,  
“shard” : 1,  
“node” : “node-221”,  
“accept\_data\_loss” : true  
\}  
\}\]  
\}

修复副本分片  
POST [http://localhost:9200/\_cluster/reroute][http_localhost_9200_cluster_reroute]  
\{  
“commands”: \[\{  
“allocate\_replica”: \{  
“index”: “.monitoring-kibana-2-2019.06.27”,  
“shard”: 0,  
“node”: “node2”,  
“allow\_primary”: true  
\}  
\}\]  
\}

## 5. 索引操作 ##

关闭指定索引  
POST [http://localhost:9200/index\_name/\_close][http_localhost_9200_index_name_close]

显示索引的别名信息、过滤器和路由信息。?v是输出表头。  
GET [http://localhost:9200/\_cat/aliases?v][http_localhost_9200_cat_aliases_v]

查看索引或集群的文档数量。  
GET [http://localhost:9200/\_cat/count?v][http_localhost_9200_cat_count_v]  
GET [http://localhost:9200/\_cat/count/books?v][http_localhost_9200_cat_count_books_v]

查看索引信息，包括：健康状态、索引开关状态、分片数、副本数、文档数量、标记为删除的文档数量、占用的存储空间、索引的唯一标识等。  
GET [http://localhost:9200/\_cat/indices?v][http_localhost_9200_cat_indices_v]  
GET [http://localhost:9200/\_cat/indices/forum?v][http_localhost_9200_cat_indices_forum_v]

## 6.其他查询 ##

当你不知道有那些属性可以查看时：  
localhost:9200/\_cat/，会返回可以查看的属性

查看正在执行的任务列表。  
GET [http://localhost:9200/\_cat/pending\_tasks?v][http_localhost_9200_cat_pending_tasks_v]

查看索引分片恢复进度。  
GET [http://localhost:9200/\_cat/recovery?v][http_localhost_9200_cat_recovery_v]

查看集群中的快照库。  
GET [http://localhost:9200/\_cat/repositories?v][http_localhost_9200_cat_repositories_v]

查看集群每个节点的线程池统计信息。  
GET [http://localhost:9200/\_cat/thread\_pool?v][http_localhost_9200_cat_thread_pool_v]

查看索引的segment信息，注意，索引数据实际上是以一个个segment的方式进行存储的。  
GET [http://localhost:9200/\_cat/segments?v][http_localhost_9200_cat_segments_v]

## 综合应用1：ELASTICSEARCH 中暂时移除一个节点 ##

除了\_name 之外, 还可以用\_ip、\_host进行匹配

步骤1：  
PUT [http://localhost:9200/\_cluster/settings][http_localhost_9200_cluster_settings]  
\{  
“transient” : \{  
“cluster.routing.allocation.exclude.\_name” : “node-2”  
\}  
\}

步骤2：  
step1 配置完成以后，我们就会看到shard在集群中开始迁移，待迁移完成以后，对node-2进行处理

步骤3：  
PUT [http://localhost:9200/\_cluster/settings][http_localhost_9200_cluster_settings]  
\{  
“transient” : \{  
“cluster.routing.allocation.exclude.\_name” : “”  
\}  
\}  
只要让\_name匹配不到在用的node即可

## 综合应用2：elasticsearch安全重启节点的方法 ##

假如要升级或重启节点192.168.1.8

第一步：先暂停集群的shard自动均衡。  
PUT [http://localhost:9200/\_cluster/settings][http_localhost_9200_cluster_settings]  
\{  
“transient” : \{  
“cluster.routing.allocation.enable” : “none”  
\}  
\}

第二步：shutdown你要升级的节点  
POST [http://192.168.1.8:9200/\_cluster/nodes/node-2/\_shutdown][http_192.168.1.8_9200_cluster_nodes_node-2_shutdown]

第三步：升级重启该节点，并确认该节点重新加入到了集群中  
GET [http://localhost:9200/\_cat/health][http_localhost_9200_cat_health]  
GET [http://localhost:9200/\_cat/nodes][http_localhost_9200_cat_nodes]

第四步：重复2-3步，升级重启其它要升级的节点。

第五步：重新启动集群的shard均衡  
PUT [http://localhost:9200/\_cluster/settings][http_localhost_9200_cluster_settings]  
\{  
“transient” : \{  
“cluster.routing.allocation.enable” : “all”  
\}  
\}

通过集群的状态和恢复进程监控集群是否可用  
GET [http://localhost:9200/\_cat/health][http_localhost_9200_cat_health]  
GET [http://localhost:9200/\_cat/recovery][http_localhost_9200_cat_recovery]

参考：  
[https://blog.51cto.com/xpleaf/2329042][https_blog.51cto.com_xpleaf_2329042]  
[https://www.cnblogs.com/leeSmall/p/9220535.html][https_www.cnblogs.com_leeSmall_p_9220535.html]

[http_localhost_9200_shutdown]: http://localhost:9200/_shutdown
[http_localhost_9200_cluster_nodes_shutdown]: http://localhost:9200/_cluster/nodes/_shutdown
[http_localhost_9200_cluster_nodes_all_shutdown]: http://localhost:9200/_cluster/nodes/_all/_shutdown
[http_localhost_9200_cluster_settings]: http://localhost:9200/_cluster/settings
[http_localhost_9200_cluster_health_pretty]: http://localhost:9200/_cluster/health?pretty
[http_localhost_9200_cat_health_v]: http://localhost:9200/_cat/health?v
[http_localhost_9200_cluster_stats_pretty]: http://localhost:9200/_cluster/stats?pretty
[http_localhost_9200_cluster_nodes_node-2_shutdown]: http://localhost:9200/_cluster/nodes/node-2/_shutdown
[http_localhost_9200_cluster_nodes_nodeId1_nodeId2_shutdown]: http://localhost:9200/_cluster/nodes/nodeId1,nodeId2/_shutdown
[http_localhost_9200_cluster_nodes_local_shutdown_delay_10s]: http://localhost:9200/_cluster/nodes/_local/_shutdown?delay=10s
[http_localhost_9200_nodes_process_pretty]: http://localhost:9200/_nodes/process?pretty
[http_localhost_9200_nodes_node1_process_pretty]: http://localhost:9200/_nodes/node1/process?pretty
[http_localhost_9200_cat_nodes_v]: http://localhost:9200/_cat/nodes?v
[http_localhost_9200_cat_plugins_v]: http://localhost:9200/_cat/plugins?v
[http_localhost_9200_cat_allocation_v]: http://localhost:9200/_cat/allocation?v
[http_localhost_9200_cat_shards_v]: http://localhost:9200/_cat/shards?v
[http_localhost_9200_cat_shards_h_index_shard_prirep_state_unassigned.reason]: http://localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason
[http_localhost_9200_cluster_allocation_explain]: http://localhost:9200/_cluster/allocation/explain
[http_localhost_9200_cluster_reroute]: http://localhost:9200/_cluster/reroute
[http_localhost_9200_index_name_close]: http://localhost:9200/index_name/_close
[http_localhost_9200_cat_aliases_v]: http://localhost:9200/_cat/aliases?v
[http_localhost_9200_cat_count_v]: http://localhost:9200/_cat/count?v
[http_localhost_9200_cat_count_books_v]: http://localhost:9200/_cat/count/books?v
[http_localhost_9200_cat_indices_v]: http://localhost:9200/_cat/indices?v
[http_localhost_9200_cat_indices_forum_v]: http://localhost:9200/_cat/indices/forum?v
[http_localhost_9200_cat_pending_tasks_v]: http://localhost:9200/_cat/pending_tasks?v
[http_localhost_9200_cat_recovery_v]: http://localhost:9200/_cat/recovery?v
[http_localhost_9200_cat_repositories_v]: http://localhost:9200/_cat/repositories?v
[http_localhost_9200_cat_thread_pool_v]: http://localhost:9200/_cat/thread_pool?v
[http_localhost_9200_cat_segments_v]: http://localhost:9200/_cat/segments?v
[http_192.168.1.8_9200_cluster_nodes_node-2_shutdown]: http://192.168.1.8:9200/_cluster/nodes/node-2/_shutdown
[http_localhost_9200_cat_health]: http://localhost:9200/_cat/health
[http_localhost_9200_cat_nodes]: http://localhost:9200/_cat/nodes
[http_localhost_9200_cat_recovery]: http://localhost:9200/_cat/recovery
[https_blog.51cto.com_xpleaf_2329042]: https://blog.51cto.com/xpleaf/2329042
[https_www.cnblogs.com_leeSmall_p_9220535.html]: https://www.cnblogs.com/leeSmall/p/9220535.html