Spring Boot实战分页查询附近的人: Redis+GeoHash+Lua

Love The Way You Lie 2022-11-25 13:24 137阅读 0赞

## 前言 ##

> 最近在做社交的业务，用户进入首页后需要查询附近的人；
> 
> 项目状况：前期尝试业务阶段；
> 
> 特点：
> 
>  *  快速实现（不需要做太重，满足初期推广运营即可）
>  *  快速投入市场去运营
> 
> 收集用户的经纬度：
> 
>  *  用户在每次启动时将当前的地理位置（经度，维度）上报给后台
> 
> 提到附近的人，脑海中首先浮现特点：
> 
>  *  需要记录每位用户的经纬度
>  *  查询当前用户附近的人，搜索在N公里内用户

## 架构设计 ##

*  时序图  
    ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTI4MjkxMjQ_size_16_color_FFFFFF_t_70_pic_center]
 *  技术实现方案
    
    SpringBoot
    
    Redis(version>=3.2)

## Redis原生命令实现 ##

*  存入用户的经纬度
    
     *  geoadd 用于存储指定的地理空间位置，可以将一个或多个经度(longitude)、纬度(latitude)、位置名称(member)添加到指定的 key 中
     *  命令格式:
        
            GEOADD key longitude latitude member [longitude latitude member ...]
     *  模拟五个用户存入经纬度，redis客户端执行如下命令：
        
            GEOADD zhgeo 116.48105 39.996794 zhangsan
            GEOADD zhgeo 116.514203 39.905409 lisi
            GEOADD zhgeo 116.489033 40.007669 wangwu
            GEOADD zhgeo 116.562108 39.787602 sunliu
            GEOADD zhgeo 116.334255 40.027400 zhaoqi
     *  通过redis客户端查看效果：  
        ![在这里插入图片描述][20200809180623685.png_pic_center]
 *  查找距当前用户由近到远附近100km用户
    
     *  georadiusbymember可以找出位于指定范围内的元素，georadiusbymember 的中心点是由给定的位置元素决定的
     *  命令格式:
        
            GEORADIUSBYMEMBER key member radius m|km|ft|mi [WITHCOORD] [WITHDIST] [WITHHASH] [COUNT count] [ASC|DESC] [STORE key] [STOREDIST key]
     *  模拟查找100km里距离sunliu由近到远五个人
        
            georadiusbymember zhgeo sunliu 100 km asc count 5
     *  命令执行效果如下：  
        ![在这里插入图片描述][20200809180856647.png_pic_center]
 *  如何实现分页查询那？
    
     *  每次分页查询的请求都计算一次然后拿到程序中在取相应的分页数据，优缺点：
        
         *  优点：实现简单，无需额外的存储空间
         *  缺点：当用户量大时，很显然不仅效率低，而且容易把程序的内存搞溢出
     *  经过查找发现redis的github官网给出了分页Issues(参考:[Will the GEORADIUS support pagination?][Will the GEORADIUS support pagination])，解决方案如下：
        
         *  利用GEORADIUSBYMEMBER 命令中的 STOREDIST 将排好序的数据存入一个Zset集合中，以后分页查直接从Zset
         *  命令如下：
            
                georadiusbymember zhgeo sunliu 100 km asc count 5 storedist sunliu
         *  有序集合效果如下:  
            ![在这里插入图片描述][20200809180939803.png_pic_center]
         *  以后分页查询命令:
            
                //首先删除本身元素
                zrem sunliu sunliu
                //分页查找元素（在此以:查找第1页，每页数量是3为例）
                zrange sunliu 0 2 withscores
         *  效果如下：  
            ![在这里插入图片描述][20200809181007105.png_pic_center]

## 代码实现 ##

*  完整代码（GitHub，欢迎大家**Star,Fork,Watch**)
    
    [**https://github.com/dangnianchuntian/springboot**][https_github.com_dangnianchuntian_springboot]
 *  主要代码展示
    
     *  Controller
    
        /* * Copyright (c) 2020. zhanghan_java@163.com All Rights Reserved. * 项目名称：Spring Boot实战分页查询附近的人: Redis+GeoHash+Lua * 类名称：GeoController.java * 创建人：张晗 * 联系方式：zhanghan_java@163.com * 开源地址: https://github.com/dangnianchuntian/springboot * 博客地址: https://zhanghan.blog.csdn.net */
        
        package com.zhanghan.zhnearbypeople.controller;
        
        import org.springframework.beans.factory.annotation.Autowired;
        import org.springframework.validation.annotation.Validated;
        import org.springframework.web.bind.annotation.RequestBody;
        import org.springframework.web.bind.annotation.RequestMapping;
        import org.springframework.web.bind.annotation.RequestMethod;
        import org.springframework.web.bind.annotation.RestController;
        
        import com.zhanghan.zhnearbypeople.controller.request.ListNearByPeopleRequest;
        import com.zhanghan.zhnearbypeople.controller.request.PostGeoRequest;
        import com.zhanghan.zhnearbypeople.service.GeoService;
        
        @RestController
        public class GeoController { 
        
            @Autowired
            private GeoService geoService;
        
            /** * 记录用户地理位置 */
            @RequestMapping(value = "/post/geo", method = RequestMethod.POST)
            public Object postGeo(@RequestBody @Validated PostGeoRequest postGeoRequest) { 
                return geoService.postGeo(postGeoRequest);
            }
        
            /** * 分页查询当前用户附近的人 */
            @RequestMapping(value = "/list/nearby/people", method = RequestMethod.POST)
            public Object listNearByPeople(@RequestBody @Validated ListNearByPeopleRequest listNearByPeopleRequest) { 
                return geoService.listNearByPeople(listNearByPeopleRequest);
            }
        
        }
 *  service
    
        /* * Copyright (c) 2020. zhanghan_java@163.com All Rights Reserved. * 项目名称：Spring Boot实战分页查询附近的人: Redis+GeoHash+Lua * 类名称：GeoServiceImpl.java * 创建人：张晗 * 联系方式：zhanghan_java@163.com * 开源地址: https://github.com/dangnianchuntian/springboot * 博客地址: https://zhanghan.blog.csdn.net */
        
        package com.zhanghan.zhnearbypeople.service.impl;
        
        import java.util.ArrayList;
        import java.util.List;
        import java.util.Set;
        
        import org.slf4j.Logger;
        import org.slf4j.LoggerFactory;
        import org.springframework.beans.factory.annotation.Autowired;
        import org.springframework.beans.factory.annotation.Value;
        import org.springframework.data.geo.Point;
        import org.springframework.data.redis.connection.RedisGeoCommands;
        import org.springframework.data.redis.core.RedisTemplate;
        import org.springframework.data.redis.core.ZSetOperations;
        import org.springframework.stereotype.Service;
        
        import com.zhanghan.zhnearbypeople.controller.request.ListNearByPeopleRequest;
        import com.zhanghan.zhnearbypeople.controller.request.PostGeoRequest;
        import com.zhanghan.zhnearbypeople.dto.NearByPeopleDto;
        import com.zhanghan.zhnearbypeople.service.GeoService;
        import com.zhanghan.zhnearbypeople.util.RedisLuaUtil;
        import com.zhanghan.zhnearbypeople.util.wrapper.WrapMapper;
        
        @Service
        public class GeoServiceImpl implements GeoService { 
        
            private static Logger logger = LoggerFactory.getLogger(GeoServiceImpl.class);
        
            @Autowired
            private RedisTemplate<String, Object> objRedisTemplate;
        
            @Value("${zh.geo.redis.key:zhgeo}")
            private String zhGeoRedisKey;
        
            @Value("${zh.geo.zset.redis.key:zhgeozset:}")
            private String zhGeoZsetRedisKey;
        
            /** * 记录用户访问记录 */
            @Override
            public Object postGeo(PostGeoRequest postGeoRequest) { 
        
                //对应redis原生命令:GEOADD zhgeo 116.48105 39.996794 zhangsan
                Long flag = objRedisTemplate.opsForGeo().add(zhGeoRedisKey, new RedisGeoCommands.GeoLocation<>(postGeoRequest
                        .getCustomerId(), new Point(postGeoRequest.getLatitude(), postGeoRequest.getLongitude())));
        
                if (null != flag && flag > 0) { 
                    return WrapMapper.ok();
                }
        
                return WrapMapper.error();
            }
        
            /** * 分页查询附近的人 */
            @Override
            public Object listNearByPeople(ListNearByPeopleRequest listNearByPeopleRequest) { 
        
                String customerId = listNearByPeopleRequest.getCustomerId();
        
                String strZsetUserKey = zhGeoZsetRedisKey + customerId;
        
                List<NearByPeopleDto> nearByPeopleDtoList = new ArrayList<>();
        
                //如果是从第1页开始查,则将附近的人写入zset集合，以后页直接从zset中查
                if (1 == listNearByPeopleRequest.getPageIndex()) { 
                    List<String> scriptParams = new ArrayList<>();
                    scriptParams.add(zhGeoRedisKey);
                    scriptParams.add(customerId);
                    scriptParams.add("100");
                    scriptParams.add(RedisGeoCommands.DistanceUnit.KILOMETERS.getAbbreviation());
                    scriptParams.add("asc");
                    scriptParams.add("storedist");
                    scriptParams.add(strZsetUserKey);
        
                    //用Lua脚本实现georadiusbymember中的storedist参数
                    //对应Redis原生命令:georadiusbymember zhgeo sunliu 100 km asc count 5 storedist sunliu
                    Long executeResult = objRedisTemplate.execute(RedisLuaUtil.GEO_RADIUS_STOREDIST_SCRIPT(), scriptParams);
        
                    if (null == executeResult || executeResult < 1) { 
                        return WrapMapper.ok(nearByPeopleDtoList);
                    }
        
                    //zset集合中去除自己
                    //对应Redis原生命令:zrem sunliu sunliu
                    Long remove = objRedisTemplate.opsForZSet().remove(strZsetUserKey, customerId);
        
                }
        
                nearByPeopleDtoList = listNearByPeopleFromZset(strZsetUserKey, listNearByPeopleRequest.getPageIndex(),
                        listNearByPeopleRequest.getPageSize());
        
                return WrapMapper.ok(nearByPeopleDtoList);
        
            }
        
            /** * 分页从zset中查询指定用户附近的人 */
            private List<NearByPeopleDto> listNearByPeopleFromZset(String strZsetUserKey, Integer pageIndex, Integer pageSize) { 
        
                Integer startPage = (pageIndex - 1) * pageSize;
                Integer endPage = pageIndex * pageSize - 1;
                List<NearByPeopleDto> nearByPeopleDtoList = new ArrayList<>();
                //对应Redis原生命令:zrange key 0 2 withscores
                Set<ZSetOperations.TypedTuple<Object>> zsetUsers = objRedisTemplate.opsForZSet()
                        .rangeWithScores(strZsetUserKey, startPage,
                                endPage);
        
                for (ZSetOperations.TypedTuple<Object> zsetUser : zsetUsers) { 
                    NearByPeopleDto nearByPeopleDto = new NearByPeopleDto();
                    nearByPeopleDto.setCustomerId(zsetUser.getValue().toString());
                    nearByPeopleDto.setDistance(zsetUser.getScore());
                    nearByPeopleDtoList.add(nearByPeopleDto);
                }
        
                return nearByPeopleDtoList;
            }
        }
 *  RedisLuaUtil
    
        /* * Copyright (c) 2020. zhanghan_java@163.com All Rights Reserved. * 项目名称：Spring Boot实战分页查询附近的人: Redis+GeoHash+Lua * 类名称：RedisLuaUtil.java * 创建人：张晗 * 联系方式：zhanghan_java@163.com * 开源地址: https://github.com/dangnianchuntian/springboot * 博客地址: https://zhanghan.blog.csdn.net */
        
        package com.zhanghan.zhnearbypeople.util;
        
        
        import org.springframework.data.redis.core.script.DigestUtils;
        import org.springframework.data.redis.core.script.RedisScript;
        
        public class RedisLuaUtil { 
        
            private static final RedisScript<Long> GEO_RADIUS_STOREDIST_SCRIPT;
        
            public static RedisScript<Long> GEO_RADIUS_STOREDIST_SCRIPT() { 
                return GEO_RADIUS_STOREDIST_SCRIPT;
            }
        
            static { 
                StringBuilder sb = new StringBuilder();
                sb.append("return redis.call('georadiusbymember',KEYS[1],KEYS[2],KEYS[3],KEYS[4],KEYS[5],KEYS[6],KEYS[7])");
                GEO_RADIUS_STOREDIST_SCRIPT = new RedisScriptImpl<>(sb.toString(), Long.class);
            }
        
            private static class RedisScriptImpl<T> implements RedisScript<T> { 
                private final String script;
                private final String sha1;
                private final Class<T> resultType;
        
                public RedisScriptImpl(String script, Class<T> resultType) { 
                    this.script = script;
                    this.sha1 = DigestUtils.sha1DigestAsHex(script);
                    this.resultType = resultType;
                }
        
                @Override
                public String getSha1() { 
                    return sha1;
                }
        
                @Override
                public Class<T> getResultType() { 
                    return resultType;
                }
        
                @Override
                public String getScriptAsString() { 
                    return script;
                }
            }
        }

## 测试 ##

*  模拟用户上传地理位置进行存储
    
     *  进行请求  
        ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTI4MjkxMjQ_size_16_color_FFFFFF_t_70_pic_center 1]
     *  查看效果  
        ![在这里插入图片描述][20200809181457887.png_pic_center]  
        ![在这里插入图片描述][2020080918155012.png_pic_center]
 *  模拟用户sunliu查找附近100km的人，按3条一分页进行查询
    
     *  进行请求  
        ![在这里插入图片描述][watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTI4MjkxMjQ_size_16_color_FFFFFF_t_70_pic_center 2]

## 总结 ##

*  亮点：
    
     *  分页实现思路：将geo集合中的数据按距离由近到远筛选好后，通过storedist放入一个新的Zset集合
     *  redisTemplate没有针对原生命令georadiusbymember的storedist参数实现，灵活**运用Lua脚本去实现**
 *  geo集合在亿级别以内的数据量没有问题，当超过亿后需要根据产品需要对Redis中的大值进行拆分，比如按照地域进行拆分等
 *  有了地理位置，自己正在研究如何通过经纬度绘制出自己的运动路线，验证出来后与大家共享

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTI4MjkxMjQ_size_16_color_FFFFFF_t_70_pic_center]: /images/20221124/9fbeb77dc629486789b6cb3b82f8b4c2.png
[20200809180623685.png_pic_center]: /images/20221124/4e06698220f74a2a83f91de15c022a88.png
[20200809180856647.png_pic_center]: /images/20221124/1800ddc21b1f40a9811727abaffef8e8.png
[Will the GEORADIUS support pagination]: https://github.com/redis/redis/issues/3019
[20200809180939803.png_pic_center]: /images/20221124/433213c0728148389da07b9c9f871532.png
[20200809181007105.png_pic_center]: /images/20221124/f6ce59b334224b3fbda1ccaf6c5fa88d.png
[https_github.com_dangnianchuntian_springboot]: https://github.com/dangnianchuntian/springboot
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTI4MjkxMjQ_size_16_color_FFFFFF_t_70_pic_center 1]: /images/20221124/e243773bfaa34a0382cc4fe96ad3ca8f.png
[20200809181457887.png_pic_center]: /images/20221124/18b3e0a52da74ff985d4c8c83c94ea2f.png
[2020080918155012.png_pic_center]: /images/20221124/00e46890714b4396aa96e578277e6e3c.png
[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTI4MjkxMjQ_size_16_color_FFFFFF_t_70_pic_center 2]: /images/20221124/2256ddf5e71a4a10b1a01b74a2a91fbe.png