elasticsearch 之 histogram 直方图聚合

青旅半醒 2024-03-31 10:41 12阅读 0赞

## 1. 简介 ##

`直方图聚合`是一种基于多桶值聚合，可从文档中提取的`数值`或`数值范围值`来进行聚合。它可以对参与聚合的值来动态的生成固定大小的桶。

## 2. bucket\_key如何计算 ##

假设我们有一个值是`32`，并且桶的大小是`5`，那么32四舍五入后变成30，因此文档将落入与键30关联的存储桶中。`下面的算式可以精确的确定每个文档的归属桶`

`bucket_key = Math.floor((value - offset) / interval) * interval + offset`

1.  `offset：`的值默认是从`0`开始。并且offset的值必须在`[0, interval)`之间。且需要是一个`正数`。
2.  `value：`值的参与计算的值，比如某个文档中的价格字段等。

## 3. 有一组数据，如何确定是落入到那个桶中 ##

> 此处是我自己的一个理解，如果错误欢迎指出。

**存在的数据： \[3, 8, 15\]**  
offset = 0  
interval = 5

那么可能会分成如下几个桶 `[0,5) [5,10) [10, 15) [15,+∞)`

1.  数字3落入的桶 buket\_key= `Math.floor((3 - 0) / 5) * 5 + 0 = 0`，即落入`[0,5)`这个桶中
2.  数字8落入的桶 buket\_key= `Math.floor((8 - 0) / 5) * 5 + 0 = 5`，即落入`[5,10)`这个桶中
3.  数字15落入的桶 buket\_key= `Math.floor((15 - 0) / 5) * 5 + 0 = 15`，即落入`[15,+∞)`这个桶中

## 4、需求 ##

我们有一组`api`响应时间数据，根据这组数据进行`histogram`聚合统计

### 4.1 准备mapping ###

PUT /index_api_response_time
    {
      "settings": {
        "number_of_shards": 1
      },
      "mappings": {
        "properties": {
          "id": {
            "type": "long"
          },
          "api": {
            "type": "keyword"
          },
          "response_time": {
            "type": "integer"
          }
        }
      }
    }

此处的`mapping`比较简单，就3个字段`id`，`api`和`response_time`。

### 4.2 准备数据 ###

PUT /index_api_response_time/_bulk
    {"index":{"_id":1}}
    {"api":"/user/infos","response_time": 3}
    {"index":{"_id":2}}
    {"api":"/user/add"}
    {"index":{"_id":3}}
    {"api":"/user/update","response_time": 8}
    {"index":{"_id":4}}
    {"api":"/user/list","response_time": 15}
    {"index":{"_id":5}}
    {"api":"/user/export","response_time": 30}
    {"index":{"_id":6}}
    {"api":"/user/detail","response_time": 32}

此处先记录 `id=2`的数据，这个是没有`response_time`的，后期聚合时额外处理。

## 5、histogram聚合操作 ##

### 5.1、根据response\_time聚合，间隔为5 ###

#### 5.1.1 dsl ####

GET /index_api_response_time/_search
    {
      "size": 0,
      "aggs": {
        "agg_01": {
          "histogram": {
            "field": "response_time",
            "interval": 5
          }
        }
      }
    }

#### 5.1.2 java代码 ####

@Test
    @DisplayName("根据response_time聚合，间隔为5")
    public void test01() throws IOException {
        
        SearchRequest request = SearchRequest.of(search ->
                search
                        .index("index_api_response_time")
                        .size(0)
                        .aggregations("agg_01", agg -> agg.histogram(histogram -> histogram.field("response_time")
                        .interval(5D))));
        System.out.println("request: " + request);
        SearchResponse<String> response = client.search(request, String.class);
        System.out.println("response: " + response);
    }

#### 5.1.3 运行结果 ####

![运行结果][5977776ca6e2449090b2c8d494e6b138.png]

### 5.2 在5.1基础上聚合出每个桶总的响应时间 ###

`此处聚合一下是为了结合已有的数据，看看每个数据是否落入到了相应的桶中`

#### 5.2.1 dsl ####

GET /index_api_response_time/_search
    {
      "size": 0,
    
      "aggs": {
        "agg_01": {
          "histogram": {
            "field": "response_time",
            "interval": 5
          },
          "aggs": {
            "agg_sum": {
              "sum": {
                "field": "response_time"
              }
            }
          }
        }
      }
    }

#### 5.2.2 java代码 ####

@Test
    @DisplayName("在test01基础上聚合出每个桶总的响应时间")
    public void test02() throws IOException {
        
        SearchRequest request = SearchRequest.of(search ->
                search
                        .index("index_api_response_time")
                        .size(0)
                        .aggregations("agg_01", agg ->
                                agg.histogram(histogram -> histogram.field("response_time").interval(5D))
                                   .aggregations("agg_sum", aggSum -> aggSum.sum(sum -> sum.field("response_time")))
                        ));
        System.out.println("request: " + request);
        SearchResponse<String> response = client.search(request, String.class);
        System.out.println("response: " + response);
    }

#### 5.2.3 运行结果 ####

![运行结果][46dee676ec6a45eb9b00ebb299286126.png]

### 5.3 每个桶中必须存在1个文档的结果才返回-min\_doc\_count ###

`从5.1中的结果我们可以知道，不管桶中是否存在数据，我们都返回了，即返回了很多空桶。 简单理解就是返回的 桶中存在 doc_count=0 的数据，此处我们需要将这个数据不返回`

#### 5.3.1 dsl ####

GET /index_api_response_time/_search
    {
      "size": 0,
    
      "aggs": {
        "agg_01": {
          "histogram": {
            "field": "response_time",
            "interval": 5,
            "min_doc_count": 1
          }
        }
      }
    }

#### 5.3.2 java代码 ####

@Test
    @DisplayName("每个桶中必须存在1个文档的结果才返回-min_doc_count")
    public void test03() throws IOException {
        
        SearchRequest request = SearchRequest.of(search ->
                search
                        .index("index_api_response_time")
                        .size(0)
                        .aggregations("agg_01", agg -> agg.histogram(
                                histogram -> histogram.field("response_time").interval(5D).minDocCount(1)
                                )
                        )
        );
        System.out.println("request: " + request);
        SearchResponse<String> response = client.search(request, String.class);
        System.out.println("response: " + response);
    }

#### 5.3.3 运行结果 ####

![运行结果][db874b7ea9c245eb9024af24efffca97.png]

### 5.4 补充空桶数据-extended\_bounds ###

`这个是什么意思？假设我们通过 response_time >= 10 进行过滤，并且 interval=5 那么es默认情况下就不会返回 bucket_key =0,5,10的桶，那么如果我想返回那么该如何处理呢？可以通过 extended_bounds 来实现`。  
使用`extended_bounds`时，`min_doc_count=0`时才有意义。 extended\_bounds不会过滤桶。

![extended\_bound解释][extended_bound]

#### 5.4.1 dsl ####

GET /index_api_response_time/_search
    {
      "size": 0,
      "query": {
        "range": {
          "response_time": {
            "gte": 10
          }
        }
      }, 
      "aggs": {
        "agg_01": {
          "histogram": {
            "field": "response_time",
            "interval": 5,
            "min_doc_count": 0,
            "extended_bounds": {
              "min": 0,
              "max": 50
            }
          }
        }
      }
    }

#### 5.4.2 java代码 ####

@Test
    @DisplayName("补充空桶数据-extended_bounds")
    public void test04() throws IOException {
        
        SearchRequest request = SearchRequest.of(search ->
                search
                        .index("index_api_response_time")
                        .size(0)
                        .query(query-> query.range(range -> range.field("response_time").gte(JsonData.of(10))))
                        .aggregations("agg_01", agg -> agg.histogram(
                                histogram -> histogram.field("response_time").interval(5D).minDocCount(0)
                                        .extendedBounds(bounds -> bounds.min(1D).max(50D))
                                )
                        )
        );
        System.out.println("request: " + request);
        SearchResponse<String> response = client.search(request, String.class);
        System.out.println("response: " + response);
    }

#### 5.4.3 运行结果 ####

![运行结果][ef361b2df31c4756b8b89de4f530ba80.png]

### 5.5 只展示min-max之间的桶-hard\_bounds ###

![只返回min-max之间的桶][min-max]  
此处的数据:

PUT /index_api_response_time/_bulk
    {"index":{"_id":1}}
    {"api":"/user/infos","response_time": 3}
    {"index":{"_id":2}}
    {"api":"/user/add"}
    {"index":{"_id":3}}
    {"api":"/user/update","response_time": 8}
    {"index":{"_id":4}}
    {"api":"/user/list","response_time": 15}
    {"index":{"_id":5}}
    {"api":"/user/export","response_time": 25}
    {"index":{"_id":6}}
    {"api":"/user/detail","response_time": 32}

#### 5.5.1 dsl ####

GET /index_api_response_time/_search
    {
      "size": 0,
      "query": {
        "range": {
          "response_time": {
            "gte": 10
          }
        }
      }, 
      "aggs": {
        "agg_01": {
          "histogram": {
            "field": "response_time",
            "interval": 5,
            "min_doc_count": 0,
            "hard_bounds": {
              "min": 15,
              "max": 25
            }
          },
          "aggs": {
            "a_s": {
              "sum": {
                "field": "response_time"
              }
            }
          }
        }
      }
    }

#### 5.5.2 java代码 ####

@Test
    @DisplayName("只展示min-max之间的桶-hard_bounds")
    public void test05() throws IOException {
        
        SearchRequest request = SearchRequest.of(search ->
                search
                        .index("index_api_response_time")
                        .size(0)
                        .query(query-> query.range(range -> range.field("response_time").gte(JsonData.of(10))))
                        .aggregations("agg_01", agg ->
                                agg.histogram(
                                    histogram -> histogram.field("response_time").interval(5D).minDocCount(0)
                                            .hardBounds(bounds -> bounds.min(1D).max(50D))
                                )
                                   .aggregations("a_s", sumAgg -> sumAgg.sum(sum -> sum.field("response_time")))
                        )
        );
        System.out.println("request: " + request);
        SearchResponse<String> response = client.search(request, String.class);
        System.out.println("response: " + response);
    }

#### 5.5.3 运行结果 ####

![运行结果][a8c67e3dbc194627bd4c238fd7df4f45.png]

### 5.6 排序-order ###

By default the returned buckets are sorted by their `key` ascending, though the order behaviour can be controlled using the order setting. Supports the same order functionality as the `Terms Aggregation`.

#### 5.6.1 dsl ####

#### 5.6.2 java代码 ####

@Test
    @DisplayName("排序order")
    public void test06() throws IOException {
        
        SearchRequest request = SearchRequest.of(search ->
                search
                        .index("index_api_response_time")
                        .size(0)
                        .query(query-> query.range(range -> range.field("response_time").gte(JsonData.of(10))))
                        .aggregations("agg_01", agg ->
                                agg.histogram(
                                    histogram -> histogram.field("response_time").interval(5D)
                                            .order(NamedValue.of("_count", SortOrder.Desc))
                                )
                        )
        );
        System.out.println("request: " + request);
        SearchResponse<String> response = client.search(request, String.class);
        System.out.println("response: " + response);
    }

#### 5.6.3 运行结果 ####

![运行结果][e551fcf4b26948728a36bdb579635f18.png]

### 5.7 文档中缺失聚合字段时如何处理-missing ###

![missing value][]

#### 5.7.1 dsl ####

GET /index_api_response_time/_search
    {
      "size": 0,
      "aggs": {
        "agg_01": {
          "histogram": {
            "field": "response_time",
            "interval": 5,
            "missing": 0
          }
        }
      }
    }

#### 5.7.2 java代码 ####

@Test
    @DisplayName("文档中缺失聚合字段时如何处理-missing")
    public void test07() throws IOException {
        
        SearchRequest request = SearchRequest.of(search ->
                search
                        .index("index_api_response_time")
                        .size(0)
                        .query(query-> query.range(range -> range.field("response_time").gte(JsonData.of(10))))
                        .aggregations("agg_01", agg ->
                                agg.histogram(
                                    histogram -> histogram.field("response_time").interval(5D) .missing(0D)
                                )
                        )
        );
        System.out.println("request: " + request);
        SearchResponse<String> response = client.search(request, String.class);
        System.out.println("response: " + response);
    }

#### 5.7.3 运行结果 ####

![运行结果][53be38a628294f929ac5f4c7a3ab25f2.png]

## 6、完整代码 ##

[https://gitee.com/huan1993/spring-cloud-parent/blob/master/es/es8-api/src/main/java/com/huan/es8/aggregations/bucket/HistogramAggs.java][https_gitee.com_huan1993_spring-cloud-parent_blob_master_es_es8-api_src_main_java_com_huan_es8_aggregations_bucket_HistogramAggs.java]

## 7、参考文档 ##

1.  [https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-histogram-aggregation.html][https_www.elastic.co_guide_en_elasticsearch_reference_current_search-aggregations-bucket-histogram-aggregation.html]

[5977776ca6e2449090b2c8d494e6b138.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/31/a575ef1005834650ad940b1ca27655b8.png
[46dee676ec6a45eb9b00ebb299286126.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/31/be2e048d7f434bfca645bc68c91aebb5.png
[db874b7ea9c245eb9024af24efffca97.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/31/344c442420684cb1b7e6920eb459bd47.png
[extended_bound]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/31/6bff264f92bb4f0998f1b927223d3c0d.png
[ef361b2df31c4756b8b89de4f530ba80.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/31/7c5833d6fb284c98b9151499e5526c15.png
[min-max]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/31/13e3cba0ab134c25aff2eb0198e8e88e.png
[a8c67e3dbc194627bd4c238fd7df4f45.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/31/23075db230984798864a0e5e144860ca.png
[e551fcf4b26948728a36bdb579635f18.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/31/4f7f45b9bb274e88b5341d05ec66a355.png
[missing value]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/31/abc607217041437a8b69e07aae1e0917.png
[53be38a628294f929ac5f4c7a3ab25f2.png]: https://image.dandelioncloud.cn/pgy_files/images/2024/03/31/208f786e1e594dc5974a6cd6fefc9d11.png
[https_gitee.com_huan1993_spring-cloud-parent_blob_master_es_es8-api_src_main_java_com_huan_es8_aggregations_bucket_HistogramAggs.java]: https://gitee.com/huan1993/spring-cloud-parent/blob/master/es/es8-api/src/main/java/com/huan/es8/aggregations/bucket/HistogramAggs.java
[https_www.elastic.co_guide_en_elasticsearch_reference_current_search-aggregations-bucket-histogram-aggregation.html]: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-histogram-aggregation.html