【SolrCloud原理】Solr Overseer优化启动性能

╰+哭是因爲堅強的太久メ 2022-05-04 03:56 278阅读 0赞

**一、集群信息**  
Solr版本：Solr 6.2，内核修改较多  
集群规模：30个节点，每个节点5个SolrServer实例；1000 coll \* 10 shard \* 3 replica = 30k core

**二、优化前集群启动状况**

1.  重启成功还是失败？失败，索引无法对外服务。
2.  重启时长多长？重启失败后需维护人员手工干预，每个实例重启约8分钟。WTF~~
3.  日志信息汇总分析（Solr和ZK）从ZK看到stat /solr/overseer/queue达到9W多，可见消息积压严重。 ![stat /solr/overseer/queue][stat _solr_overseer_queue]

**三、优化后集群启动状况**

1.  重启成功还是失败？成功！！！
2.  重启时长多长？约30分钟左右，集群所有索引状态恢复正常。
3.  日志信息汇总分析（Solr和ZK）stat /solr/overseer/queue最多只到达16000个子节点。注意Overseer中的[STATE\_UPDATE\_MAX\_QUEUE=20000可能导致队列占满异常][STATE_UPDATE_MAX_QUEUE_20000]的情况，已经适度调大，并在ZkDistributedQueue对写入ZK增加重试逻辑。

**四、Performance Benchmark**

1.  3W Core无数据 —— 重启耗时统计：重启Solr集群约30分钟，所有shard状态正常。
2.  每个实例500M —— 重启耗时统计：重启Solr集群约40分钟，所有shard状态正常。
3.  每个实例1G —— 重启耗时统计：重启Solr集群约40分钟，所有shard状态正常。
4.  每个实例10G —— 重启耗时统计：重启Solr集群约41分钟，所有shard状态正常。
5.  每个实例100G —— 重启耗时统计：重启Solr集群约68分钟，所有shard状态正常。
6.  每个实例300G —— 重启耗时统计：重启Solr集群约85分钟，所有shard状态正常。

**五、社区Jira分析与总结**  
[Overseer can become the bottleneck in very large clusters][]  
这是个大单子，还没有彻底解决，一共包含了4个子单，其中3张子单已经resolved。其中本次优化主要合入Solr 6.2的是以下四张单子：

1.  **Improve stability and startup performance of SolrCloud with thousands of collections**  
    [https://issues.apache.org/jira/browse/SOLR-7191][https_issues.apache.org_jira_browse_SOLR-7191]  
    **主要优化点：**  
    a. 将coreZkRegister的实现类从newMDCAwareCachedThreadPool改成了newMDCAwareFixedThreadPool，因此需要将线程数作为参数传入；  
    b. SolrCore，SolrCores，CoreDescriptor支持排序
2.  **Better ZkStateWriter batching**  
    [https://issues.apache.org/jira/browse/SOLR-10524][https_issues.apache.org_jira_browse_SOLR-10524]  
    **主要优化点：**  
    a. 批量读取ZK中Overseer queue的消息  
    b. 对Msg排序，逐个处理移除
3.  **Optimize using cache for DistributedQueue in case of single-consumer**  
    [https://issues.apache.org/jira/browse/SOLR-10619][https_issues.apache.org_jira_browse_SOLR-10619]  
    **主要优化点：**  
    a. 添加一个flag布尔参数singleConsumer，针对该场景做优化  
    b. 优化fetch from zk的条件
4.  **Remove the usage of workqueue for Overseer**  
    [https://issues.apache.org/jira/browse/SOLR-11443][https_issues.apache.org_jira_browse_SOLR-11443]  
    **主要优化点：**  
    a. Overseer中移除了workqueue的设计，采用batch操作大大缩减了对ZK的写入压力，但仍然是单线程循序处理Overseer queue的消息。  
    b. ZkStateWriter中添加批量更新的设计，优化了flush机制：与上一次写入ZK间隔2秒或者积压10000个状态更新请求时才往ZK写入一次。这个设计减少了Overseer与ZK之间的网络通信往返次数，大幅度提高ZK写入性能。  
    c. ZkDistributedQueue中添加批量删除ZK node的remove方法，同样减少了Overseer与ZK之间的网络通信往返次数，提升了写入效率。  
    d. That is all?

**References**  
[https://issues.apache.org/jira/browse/SOLR-5872?jql=text ~ "overseer"][https_issues.apache.org_jira_browse_SOLR-5872_jql_text _ _overseer] (Overseer的JIRA单真不少！)  
[https://issues.apache.org/jira/browse/SOLR-5872][https_issues.apache.org_jira_browse_SOLR-5872] (你想消除Overseer？！那你就去做丫~)  
[https://issues.apache.org/jira/browse/SOLR-10265][Overseer can become the bottleneck in very large clusters] (So great!!! Overseer是大集群的性能瓶颈，分析得很清楚！)  
[https://issues.apache.org/jira/browse/SOLR-5475][https_issues.apache.org_jira_browse_SOLR-5475] (多线程想法，咋没人实现尼？风险在哪里？)  
[https://blog.csdn.net/iteye\_423/article/details/82614070][https_blog.csdn.net_iteye_423_article_details_82614070]

还有两张更久远的单子：

1.  New optimized DistributedQueue implementation for overseer  
    [https://issues.apache.org/jira/browse/SOLR-6760][https_issues.apache.org_jira_browse_SOLR-6760]
2.  Improve stability and startup performance of SolrCloud with thousands of collections  
    [https://issues.apache.org/jira/browse/SOLR-7191][https_issues.apache.org_jira_browse_SOLR-7191]

**启动异常解析**  
1、Solr 6.2的PeerSync.java在高亮那行报了空指针异常:  
*【划重点】开源Solr 7.4之后对PeerSync.java优化，增加了判空逻辑*，社区相关JIRA单为[https://issues.apache.org/jira/browse/SOLR-10169][https_issues.apache.org_jira_browse_SOLR-10169]

2019-01-23 18:32:17,169 | ERROR | recoveryExecutor-3-thread-112-processing-n:189.39.172.103:21104\_solr x:coll\_test497\_shard1\_replica1 s:shard1 c:coll\_test497 r:core\_node8 | Error while trying to recover. core=coll\_test497\_shard1\_replica1:java.lang.NullPointerException  
at org.apache.solr.update.PeerSync.alreadyInSync(PeerSync.java:339)  
at org.apache.solr.update.PeerSync.sync(PeerSync.java:222)  
at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:382)  
at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:222)  
at java.util.concurrent.Executors R u n n a b l e A d a p t e r . c a l l ( E x e c u t o r s . j a v a : 511 ) a t j a v a . u t i l . c o n c u r r e n t . F u t u r e T a s k . r u n ( F u t u r e T a s k . j a v a : 266 ) a t o r g . a p a c h e . s o l r . c o m m o n . u t i l . E x e c u t o r U t i l RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.solr.common.util.ExecutorUtil RunnableAdapter.call(Executors.java:511)atjava.util.concurrent.FutureTask.run(FutureTask.java:266)atorg.apache.solr.common.util.ExecutorUtilMDCAwareThreadPoolExecutor.lambda$execute 0 ( E x e c u t o r U t i l . j a v a : 229 ) a t j a v a . u t i l . c o n c u r r e n t . T h r e a d P o o l E x e c u t o r . r u n W o r k e r ( T h r e a d P o o l E x e c u t o r . j a v a : 1149 ) a t j a v a . u t i l . c o n c u r r e n t . T h r e a d P o o l E x e c u t o r 0(ExecutorUtil.java:229) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor 0(ExecutorUtil.java:229)atjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)atjava.util.concurrent.ThreadPoolExecutorWorker.run(ThreadPoolExecutor.java:624)  
at java.lang.Thread.run(Thread.java:748)  
| org.apache.solr.common.SolrException.log(SolrException.java:159)  
2019-01-23 18:32:17,169 | INFO | recoveryExecutor-3-thread-112-processing-n:189.39.172.103:21104\_solr x:coll\_test497\_shard1\_replica1 s:shard1 c:coll\_test497 r:core\_node8 | Replay not started, or was not successful… still buffering updates. | org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:447)  
2019-01-23 18:32:17,169 | INFO | recoveryExecutor-3-thread-112-processing-n:189.39.172.103:21104\_solr x:coll\_test497\_shard1\_replica1 s:shard1 c:coll\_test497 r:core\_node8 | RecoveryStrategy has been closed | org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:480)  
2019-01-23 18:32:17,169 | INFO | recoveryExecutor-3-thread-112-processing-n:189.39.172.103:21104\_solr x:coll\_test497\_shard1\_replica1 s:shard1 c:coll\_test497 r:core\_node8 | Finished recovery process, successful=\[false\] | org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:530)

[stat _solr_overseer_queue]: /images/20220504/857fbce24c944ca08411f6a4ffa6feda.png
[STATE_UPDATE_MAX_QUEUE_20000]: https://issues.apache.org/jira/projects/SOLR/issues/SOLR-13061
[Overseer can become the bottleneck in very large clusters]: https://issues.apache.org/jira/browse/SOLR-10265
[https_issues.apache.org_jira_browse_SOLR-7191]: https://issues.apache.org/jira/browse/SOLR-7191
[https_issues.apache.org_jira_browse_SOLR-10524]: https://issues.apache.org/jira/browse/SOLR-10524
[https_issues.apache.org_jira_browse_SOLR-10619]: https://issues.apache.org/jira/browse/SOLR-10619
[https_issues.apache.org_jira_browse_SOLR-11443]: https://issues.apache.org/jira/browse/SOLR-11443
[https_issues.apache.org_jira_browse_SOLR-5872_jql_text _ _overseer]: https://issues.apache.org/jira/browse/SOLR-5872?jql=text%20~%20%22overseer%22
[https_issues.apache.org_jira_browse_SOLR-5872]: https://issues.apache.org/jira/browse/SOLR-5872
[https_issues.apache.org_jira_browse_SOLR-5475]: https://issues.apache.org/jira/browse/SOLR-5475
[https_blog.csdn.net_iteye_423_article_details_82614070]: https://blog.csdn.net/iteye_423/article/details/82614070
[https_issues.apache.org_jira_browse_SOLR-6760]: https://issues.apache.org/jira/browse/SOLR-6760
[https_issues.apache.org_jira_browse_SOLR-10169]: https://issues.apache.org/jira/browse/SOLR-10169