Elasticsearch 去重统计 cardinality去重复

た入场券 2024-04-07 12:09 82阅读 0赞

Elasticsearch 去重统计 按照deviceId 去重统计总数

相当于SQL

SELECT COUNT(DISTINCT deviceId ?) FROM log_info where userName= 'admin' and operateTime > '2020-05-18 00:00:00' and?perateTime < '2020-05-25 00:00:00'?
    
    {
    	"query": {
    		"bool": {
    			"must": [{
    				"match": {
    					"userName": "admin"
    				}
    			}, {
    				"range": {
    					"operateTime": {
    						"gt": "2020-05-18 00:00:00",
    						"lt": "2020-05-25 00:00:00"
    					}
    				}
    			}],
    			"must_not": [],
    			"should": []
    		}
    	},
    	"from": 0,
    	"size": 0,
    	"sort": [],
    	"aggs": {
    		"deviceId_aggs": {
    			"cardinality": {
    				"field": "deviceId"
    			}
    		}
    	}
    }

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0xaSEhfMjAwOA_size_16_color_FFFFFF_t_70][]

es在使用cardinality实现count(distinct)时会在准确性和及时性上做一定的取舍

可以在使用cardinality时，配置下面的参数来增加准确性，牺牲的是时间和内存

percision\_threshold : 40000

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0xaSEhfMjAwOA_size_16_color_FFFFFF_t_70 1][]

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0xaSEhfMjAwOA_size_16_color_FFFFFF_t_70]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/07/91d888369df5445f8150ca98c44529fa.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L0xaSEhfMjAwOA_size_16_color_FFFFFF_t_70 1]: https://image.dandelioncloud.cn/pgy_files/images/2024/04/07/05c33056c9784dee8d9f5ba12a3f7127.png