ClickHouse系列--Too many partitions for single INSERT block (more than 100)

绝地灬酷狼 2023-01-23 10:47 3阅读 0赞

批量插入时，这一批数据是按照天分区的，一批数据里的日期跨度为一年，也就是单次插入可能产生365个分区，导致后台异步合并数据出现问题。  
解决方案：单批次写入数据，要控制分区数量。

exception, code: 252, host: 10.58.10.243, port: 8123; Code: 252, e.displayText() = DB::Exception: Too many partitions for single INSERT block (more than 100). The limit is controlled by 'max_partitions_per_insert_block' setting. Large number of partitions is a common misconception. It will lead to severe negative performance impact, including slow server startup, slow INSERT queries and slow SELECT queries. Recommended total number of partitions for a table is under 1000..10000. Please note, that partitioning is not intended to speed up SELECT queries (ORDER BY key is sufficient to make range queries fast). Partitions are intended for data manipulation (DROP PARTITION, etc). (version 21.5.6.6 (official build))
    
    
    ru.yandex.clickhouse.except.ClickHouseException: ClickHouse exception, code: 252, host: 10.58.10.243, port: 8123; Code: 252, e.displayText() = DB::Exception: Too many partitions for single INSERT block (more than 100). The limit is controlled by 'max_partitions_per_insert_block' setting. Large number of partitions is a common misconception. It will lead to severe negative performance impact, including slow server startup, slow INSERT queries and slow SELECT queries. Recommended total number of partitions for a table is under 1000..10000. Please note, that partitioning is not intended to speed up SELECT queries (ORDER BY key is sufficient to make range queries fast). Partitions are intended for data manipulation (DROP PARTITION, etc). (version 21.5.6.6 (official build))
    
    	at ru.yandex.clickhouse.except.ClickHouseExceptionSpecifier.specify(ClickHouseExceptionSpecifier.java:58) ~[clickhouse-jdbc-0.2.4.jar:na]
    	at ru.yandex.clickhouse.except.ClickHouseExceptionSpecifier.specify(ClickHouseExceptionSpecifier.java:28) ~[clickhouse-jdbc-0.2.4.jar:na]
    	at ru.yandex.clickhouse.ClickHouseStatementImpl.checkForErrorAndThrow(ClickHouseStatementImpl.java:875) ~[clickhouse-jdbc-0.2.4.jar:na]
    	at ru.yandex.clickhouse.ClickHouseStatementImpl.sendStream(ClickHouseStatementImpl.java:851) ~[clickhouse-jdbc-0.2.4.jar:na]
    	at ru.yandex.clickhouse.ClickHouseStatementImpl.sendStream(ClickHouseStatementImpl.java:824) ~[clickhouse-jdbc-0.2.4.jar:na]
    	at ru.yandex.clickhouse.ClickHouseStatementImpl.sendStream(ClickHouseStatementImpl.java:817) ~[clickhouse-jdbc-0.2.4.jar:na]
    	at ru.yandex.clickhouse.ClickHousePreparedStatementImpl.executeBatch(ClickHousePreparedStatementImpl.java:335) ~[clickhouse-jdbc-0.2.4.jar:na]
    	at ru.yandex.clickhouse.ClickHousePreparedStatementImpl.executeBatch(ClickHousePreparedStatementImpl.java:320) ~[clickhouse-jdbc-0.2.4.jar:na]
    	at cn.java4all.core.consumer.ClickHouseConsumer.onBulkMessage(ClickHouseConsumer.java:87) ~[classes/:na]
    	at cn.java4all.core.consumer.AbstractConsumer.doConsume(AbstractConsumer.java:69) [classes/:na]
    	at cn.fraudmetrix.module.kafka.consumer.ConcurrentPoller.lambda$handle$1(ConcurrentPoller.java:98) [module-kafka-3.2.3-20210708.074058-50.jar:na]
    	at java.util.concurrent.CompletableFuture$AsyncRun.run$$$capture(CompletableFuture.java:1626) ~[na:1.8.0_211]
    	at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java) ~[na:1.8.0_211]
    	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_211]
    	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_211]
    	at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_211]
    Caused by: java.lang.Throwable: Code: 252, e.displayText() = DB::Exception: Too many partitions for single INSERT block (more than 100). The limit is controlled by 'max_partitions_per_insert_block' setting. Large number of partitions is a common misconception. It will lead to severe negative performance impact, including slow server startup, slow INSERT queries and slow SELECT queries. Recommended total number of partitions for a table is under 1000..10000. Please note, that partitioning is not intended to speed up SELECT queries (ORDER BY key is sufficient to make range queries fast). Partitions are intended for data manipulation (DROP PARTITION, etc). (version 21.5.6.6 (official build))
    
    	at ru.yandex.clickhouse.except.ClickHouseExceptionSpecifier.specify(ClickHouseExceptionSpecifier.java:53) ~[clickhouse-jdbc-0.2.4.jar:na]
    	... 15 common frames omitted