【实践Solr深度分页】

待我称王封你为后i 2021-11-26 14:30 221阅读 0赞

## Background ##

Solr2ES, [Solr Query with Pagination(start, rows).][Solr Query with Pagination_start_ rows_.]

## Basic Paging ##

> There are two parameters that control paging:  
> start – The starting offset into the ranked (sorted) list of documents. Defaults to 0.  
> rows – The maximum number of documents to return. Defaults to 10.  
> For example, if we add start=3 and rows=2 to the query, we should get the 4th and 5th books in the ranked document list.  
> [http://localhost:8983/solr/query][http_localhost_8983_solr_query]?  
> q=id:book\*  
> &sort=pubyear\_i desc  
> &fl=title,pubyear\_i  
> &start=3  
> &rows=2

## Deep Paging ##

> Deep paging refers to specifying a large start offset into the search results.  
> Basic paging can be inefficient with large start values since to return documents 1,000,000 through 1,000,010 in a sorted document list (only 10 documents), the search engine **must find the top 1,000,010 documents and then take the last 10 to return to the user 极其低效！！！**. Solr is smart enough to only retrieve the stored fields for the final 10 documents, but there is still the overhead of sorting the internal ids of the top 1,000,010 documents.  
> Deep paging via basic paging controls is **even more inefficient for distributed searches (深度分页在SolrCloud更加低效！！！)** since the sort values for the first 1,000,010 documents from each shard need to be returned and merged at an aggregator node in order to find the correct 10.

## 成功踩坑 ##

[Performance Problems with “Deep Paging”][Performance Problems with _Deep Paging]

> In some situations, the results of a Solr search are not destined for a simple paginated user interface.  
> 查询时指定start和rows返回大量排序数据非常低效！！！  
> When you wish to fetch a very large number of sorted results from Solr to feed into an external system, using very large values for the start or rows parameters can be very inefficient. Pagination using start and rows not only require Solr to compute (and sort) in memory all of the matching documents that should be fetched for the current page, but also all of the documents that would have appeared on previous pages.  
> While a request for start=0&rows=1000000 may be obviously inefficient because it requires Solr to maintain & sort in memory a set of 1 million documents, likewise a request for start=999000&rows=1000 is equally inefficient for the same reasons. Solr can’t compute which matching document is the 999001st result in sorted order, without first determining what the first 999000 matching sorted results are.  
> 想要计算第999001个文档，必须先去计算start=999000之前的所有文档！！！我滴天~~  
> 相当于start越大，扫过的数据量会越大！最后几乎变成全表扫描！简直完美！！！  
> If the index is distributed, which is common when running in SolrCloud mode, then 1 million documents are retrieved from each shard. For a ten shard index, ten million entries must be retrieved and sorted to figure out the 1000 documents that match those query parameters.

## 快速脱坑 ##

[Deep Paging with a Cursor][]

As an alternative to increasing the “start” parameter to request subsequent pages of sorted results, Solr supports using a “Cursor” to scan through results.

Cursors in Solr are a logical concept that doesn’t involve caching any state information on the server. Instead the sort values of the last document returned to the client are used to compute a “mark” representing a logical point in the ordered space of sort values. That “mark” can be specified in the parameters of subsequent requests to tell Solr where to continue.

The cursorMark parameter allows efficient iteration over a large result set. It works on both a single node and with distributed searches and SolrCloud mode. Using cursorMark:

>  *  sort must include a tie-breaker sort on the id field. This prevents tie-breaking by internal lucene document id (which can change).
>  *  start must be 0 for all calls including a cursorMark.
>  *  pass cursorMark=\* for the first request.
>  *  Solr will return a nextCursorMark in the response. Simply use this value for cursorMark on the next call to continue paging through the results.

First request:

> [http://localhost:8983/solr/query][http_localhost_8983_solr_query]?  
> q=id:book\*  
> &sort=pubyear\_i desc, id asc  
> &fl=title,pubyear\_i  
> &rows=5  
> &cursorMark=\*

{"response":{"numFound":12,"start":0,"docs":[
        {
          "pubyear_i":2011,
          "title":["A Dance with Dragons"]},
        {
          "pubyear_i":2005,
          "title":["A Feast for Crows"]},
        {
          "pubyear_i":2005,
          "title":["Brass Man"]},
        {
          "pubyear_i":2003,
          "title":["The Line of Polity"]},
        {
          "pubyear_i":2001,
          "title":["Gridlinked"]}]
    },
    "nextCursorMark":"AoJRfSVib29rNw=="}

For the next request, simply set cursorMark to the value we received for nextCursorMark.

> [http://localhost:8983/solr/query][http_localhost_8983_solr_query]?  
> q=id:book\*  
> &sort=pubyear\_i desc, id asc  
> &fl=title,pubyear\_i  
> &rows=5  
> &cursorMark=AoJRfSVib29rNw==

{"response":{"numFound":12,"start":0,"docs":[
        {
          "pubyear_i":2000,
          "title":["A Storm of Swords"]},
        {
          "pubyear_i":1990,
          "title":["Use of Weapons"]},
        {
          "pubyear_i":1989,
          "title":["Shadow Games"]},
        {
          "pubyear_i":1988,
          "title":["The Player of Games"]},
        {
          "pubyear_i":1987,
          "title":["Consider Phlebas"]}]
    },
    "nextCursorMark":"AoJTfCVib29rNA=="}

## Implementation Notes ##

>  *  The cursorMark parameter itself contains all the necessary state. There is no server-side state.
>  *  The start parameter returned is always 0. It’s up to the client to figure out (or remember) what the position is for display purposes.
>  *  There is no need to page to the end of the result set with cursorMark (since there is no server-side state kept). Stop where ever you want.
>  *  You know you have reached the end of a result set when you do not get back the full number of rows requested, or when the nextCursorMark returned is the same as the cursorMark you sent (at which point, no documents will be in the returned list).
>  *  Although start must always be 0, you can vary the number of rows for every call to vary the page size.  
>     ![Cursor functionality requires start=0][Cursor functionality requires start_0]
>  *  You can re-use cursorMark values, changing other things like what stored fields are returned or what fields are faceted.
>  *  A client can efficiently go back pages by remembering previous cursorMarks and re-submitting them.

## Reference ##

[Solr查询参数(分页)start与rows][Solr_start_rows]  
[https://lucene.apache.org/solr/guide/6\_6/pagination-of-results.html][https_lucene.apache.org_solr_guide_6_6_pagination-of-results.html]  
[http://yonik.com/solr/paging-and-deep-paging/][Deep Paging with a Cursor]  
[http://lucene.472066.n3.nabble.com/Why-are-cursor-mark-queries-recommended-over-regular-start-rows-combination-td4379672.html][http_lucene.472066.n3.nabble.com_Why-are-cursor-mark-queries-recommended-over-regular-start-rows-combination-td4379672.html]  
[https://lucidworks.com/post/coming-soon-to-solr-efficient-cursor-based-iteration-of-large-result-sets/][https_lucidworks.com_post_coming-soon-to-solr-efficient-cursor-based-iteration-of-large-result-sets]

[Solr Query with Pagination_start_ rows_.]: https://lucene.apache.org/solr/guide/6_6/common-query-parameters.html
[http_localhost_8983_solr_query]: http://localhost:8983/solr/query
[Performance Problems with _Deep Paging]: https://lucene.apache.org/solr/guide/6_6/pagination-of-results.html#performance-problems-with-deep-paging
[Deep Paging with a Cursor]: http://yonik.com/solr/paging-and-deep-paging/
[Cursor functionality requires start_0]: /images/20211126/1140634e826b43d3ae18b8364dad3c93.png
[Solr_start_rows]: https://blog.csdn.net/jiangchao858/article/details/53615239
[https_lucene.apache.org_solr_guide_6_6_pagination-of-results.html]: https://lucene.apache.org/solr/guide/6_6/pagination-of-results.html
[http_lucene.472066.n3.nabble.com_Why-are-cursor-mark-queries-recommended-over-regular-start-rows-combination-td4379672.html]: http://lucene.472066.n3.nabble.com/Why-are-cursor-mark-queries-recommended-over-regular-start-rows-combination-td4379672.html
[https_lucidworks.com_post_coming-soon-to-solr-efficient-cursor-based-iteration-of-large-result-sets]: https://lucidworks.com/post/coming-soon-to-solr-efficient-cursor-based-iteration-of-large-result-sets/