elasticsearch实现冷热数据分离

￡神魔★判官ぃ 2023-06-19 02:18 2阅读 0赞

本文以最新的elasticsearch-6.3.0.tar.gz为例,为了节约资源,本文将副本调为0, 无client角色

https://www.elastic.co/blog/hot-warm-architecture-in-elasticsearch-5-x

以前es2.x版本配置elasticsearch.yml 里的node.tag: hot这个配置不生效了
    
    被改成了这个
    node.attr.box_type: hot

## es架构 ##

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d3ZDA1MDE_size_16_color_FFFFFF_t_70][]

## 各节点的es配置 ##

master节点:
    [root@n1 ~]# cat /usr/local/elasticsearch/config/elasticsearch.yml 
    cluster.name: elk
    node.master: true
    node.data: false
    node.name: 192.168.2.11
    #node.attr.box_type: hot
    #node.tag: hot
    path.data: /data/es 
    path.logs: /data/log
    network.host: 192.168.2.11
    http.port: 9200
    transport.tcp.port: 9300
    transport.tcp.compress: true
    discovery.zen.ping.unicast.hosts: ["192.168.2.11"]
    cluster.routing.allocation.disk.watermark.low: 85%
    cluster.routing.allocation.disk.watermark.high: 90%
    indices.fielddata.cache.size: 10%
    indices.breaker.fielddata.limit: 30%
    http.cors.enabled: true
    http.cors.allow-origin: "*"

- client节点(这里就不配置了)
    
    node.master: false
    node.data: false

- hot节点
    [root@n2 ~]# cat /usr/local/elasticsearch/config/elasticsearch.yml 
    cluster.name: elk
    node.master: false
    node.data: true
    node.name: 192.168.2.12
    node.attr.box_type: hot
    path.data: /data/es
    network.host: 192.168.2.12
    http.port: 9200
    transport.tcp.port: 9300
    transport.tcp.compress: true
    discovery.zen.ping.unicast.hosts: ["192.168.2.11"]
    cluster.routing.allocation.disk.watermark.low: 85%
    cluster.routing.allocation.disk.watermark.high: 90%
    indices.fielddata.cache.size: 10%
    indices.breaker.fielddata.limit: 30%
    http.cors.enabled: true
    http.cors.allow-origin: "*"

- cold节点
    [root@n3 ~]# cat /usr/local/elasticsearch/config/elasticsearch.yml 
    cluster.name: elk
    node.master: false
    node.data: true
    node.name: 192.168.2.13
    node.attr.box_type: cold
    path.data: /data/es
    network.host: 192.168.2.13
    http.port: 9200
    transport.tcp.port: 9300
    transport.tcp.compress: true
    discovery.zen.ping.unicast.hosts: ["192.168.2.11"]
    cluster.routing.allocation.disk.watermark.low: 85%
    cluster.routing.allocation.disk.watermark.high: 90%
    indices.fielddata.cache.size: 10%
    indices.breaker.fielddata.limit: 30%
    http.cors.enabled: true
    http.cors.allow-origin: "*"

## 如何实现某索引数据写到指定的node?(根据节点tag即可) ##

我hot节点打了tag

node.attr.box_type: cold

创建一个template(这里我用kibana来操作es的api)
    
    PUT _template/test
    {
        "index_patterns": "test-*",
        "settings": {
            "index.number_of_replicas": "0",
            "index.routing.allocation.require.box_type": "hot"
         }
    }

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d3ZDA1MDE_size_16_color_FFFFFF_t_70 1][]

意思是`test-*`索引命名的,都将其数据放到hot节点上.

## 如何实现数据从hot节点迁移到老的cold节点? ##

以test-2018.07.05索引为例,将它从hot节点迁移到cold节点

kibana里操作:
    
    PUT /test-2018.07.05/_settings 
    { 
      "settings": { 
        "index.routing.allocation.require.box_type": "cold"
      } 
    }

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d3ZDA1MDE_size_16_color_FFFFFF_t_70 2][]

生产中可能每天,或每h,生成一个index.

test-2018.07.01
    test-2018.07.02
    test-2018.07.03
    test-2018.07.04
    test-2018.07.05
    ...

我可以写一个sh定时任务,每天晚上定时迁移数据.  
如我在hot节点只保留7天的数据,7天以前的索引我匹配到, 每天晚上执行以下迁移命令即可.

## cold节点数据保留1个月? ##

https://www.cnblogs.com/iiiiher/p/8029062.html

## 优化点: ##

1.为了提高吞吐量

path.data:/data1,/data2,/data3,/data4,/data5 可以每个目录挂一块盘

2.如果有10台hot节点,可以设置10个shards

## logstash测试 ##

input { stdin { } }
    output {
        elasticsearch { 
            index => "test-%{+YYYY.MM.dd}"
            hosts => ["192.168.2.11:9200"] 
        }
        stdout {codec => rubydebug}
    }
    
    /usr/local/logstash/bin/logstash -f logstash.yaml --config.reload.automatic

## 关于es的index template ##

关于[es的template][es_template]  
es数据入库时候都会匹配一个index template,默认匹配的是logstash这个template

template大致分成setting和mappings两部分

1.  settings主要作用于index的一些相关配置信息，如分片数、副本数，tranlog同步条件、refresh等。
2.  mappings主要是一些说明信息，大致又分为\_all、\_source、prpperties这三部分： https://elasticsearch.cn/article/335

根据index name来匹配使用哪个index template. index template属于节点范围,而非全局. 需要给某个节点单独设置index\_template(如给设置一些特有的tag).

文章来源：[https://www.cnblogs.com/iiiiher/p/9268832.html][https_www.cnblogs.com_iiiiher_p_9268832.html]

[https://elasticsearch.cn/article/6127][https_elasticsearch.cn_article_6127]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d3ZDA1MDE_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20191203211522580.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d3ZDA1MDE=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d3ZDA1MDE_size_16_color_FFFFFF_t_70 1]: https://img-blog.csdnimg.cn/20191203211601340.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d3ZDA1MDE=,size_16,color_FFFFFF,t_70
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d3ZDA1MDE_size_16_color_FFFFFF_t_70 2]: https://img-blog.csdnimg.cn/20191203211625390.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3d3ZDA1MDE=,size_16,color_FFFFFF,t_70
[es_template]: https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html
[https_www.cnblogs.com_iiiiher_p_9268832.html]: https://www.cnblogs.com/iiiiher/p/9268832.html
[https_elasticsearch.cn_article_6127]: https://elasticsearch.cn/article/6127