使用Linux命令分析Nginx日志中的PVUV

以你之姓@ 2021-07-26 11:46 374阅读 0赞

近期，公司有统计网站PVUV的想法，巧合的是，公司网站上已经安装了Nginx，分析Nginx的日志来统计PVUV简直是再简单不过了。所谓PV(访问量)，即Page View，它是页面浏览量或点击量，用户每次刷新即被计算一次。UV(独立访客)，即Unique Visitor，指访问网站的一台电脑客户端为一个访客，00:00-24:00内相同的客户端只被计算一次。IP(独立IP)，即Internet Protocol，指独立IP数，00:00-24:00内相同IP地址只被计算一次。下面是分析Nginx日志常用的一些基本命令。

1.根据访问IP统计UV

[root@cdh-master files]# awk '{print $1}' access.log | sort | uniq -c | wc -l
    55

2.统计访问URL统计PV

[root@cdh-master files]# awk '{print $7}' access.log | wc -l
    23703

3.查询访问最频繁的URL

[root@cdh-master files]# awk '{print $7}' access.log | sort | uniq -c | sort -n -k 1 -r | more
       1266 /service/captcha
       1046 /api/lh-system/dict/enumDict?code=auditStatus
        939 /backend/oauth/token
        822 /api/lh-system/dict/enumDict?code=lockStatus
        802 /static/js/vendor.e4ef0e9a82c50cc8468b.js
        798 /static/js/app.5820bd1c4e778787d807.js
        797 /static/js/manifest.5f19f0f424e5074bc361.js

4.查询访问最频繁的IP

[root@cdh-master files]# awk '{print $1}' access.log | sort | uniq -c | sort -n -k 1 -r | more
      11726 10.135.254.241
       3719 10.135.66.67
       1648 10.135.198.21
       1488 10.135.252.64
        951 10.135.198.156

5.根据时间段统计查看日志

[root@cdh-master files]# cat access.log | sed -n '/19\/May\/2020:03/,/19\/May\/2020:04/p' | more
    10.135.254.241 - - [19/May/2020:03:35:23 +0800] "GET /index.html HTTP/1.1" 200 808 "-" "Mozilla/5.0 (Linux; U; Android 8.1.0; zh-CN; EML-AL00 Build/HUAWEIEML-AL00) AppleWe
    bKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/57.0.2987.108 baidu.sogo.uc.UCBrowser/11.9.4.974 UWS/2.13.1.48 Mobile Safari/537.36 AliApp(DingTalk/4.5.11) com.alibaba.
    android.rimet/10487439 Channel/227200 language/zh-CN" "-"

6.统计IP访问个数（和根据访问IP统计UV一样）

[root@cdh-master files]# cat access.log | awk '{ips[$1]+=1} END{for(ip in ips) print ips[ip],ip}' | sort -nr | wc -l
    55

7.查看3点-6点之间的Ip访问个数

[root@cdh-master files]# grep "2020:0[3-6]" access.log | awk '{ips[$1]+=1} END{for(ip in ips) print ips[ip],ip}' | sort -nr | wc -l
    6

8.查看3点-6点之间的ip访问数，并且查询出访问数>=200的ip.

[root@cdh-master files]# grep "2020:0[3-6]" access.log | awk '{ips[$1]+=1}END{for(ip in ips) if(ips[ip]>=200) print ips[ip],ip}' | sort -nr
    353 10.135.254.241

10.获取每分钟的请求数量，输出成csv文件

cat access.log | awk '{print substr($4,14,5)}' | uniq -c | awk '{print $2","$1}' > access.csv

11.获取最耗时的请求时间、url、耗时，前10名, 可以修改后面的数字获取更多，不加则获取全部

[root@cdh-master files]# cat access.log | awk '{print $4,$7,$NF}' | awk -F '"' '{print $1,$2,$3}' | sort -k3 -rn | head -10
    [19/May/2020:11:00:43 /service/captcha  223.104.189.25 
    [19/May/2020:11:00:43 /service/authorize?response_type=code&client_id=lh_platform&redirect_uri=http:%2F%2Fucenter.sdland-sea.com%2F%23%2FcodeCallback%3Fredirect_uri%3Dhttp%253A%252F%252Fucenter.sdland-sea.com%252F%2523%252F  223.104.189.25 
    [19/May/2020:11:00:34 /static/js/3.1894892101687adc4f4b.js  223.104.189.25 
    [19/May/2020:11:00:31 /service/captcha  223.104.189.25 
    [19/May/2020:11:00:31 /service/authorize?response_type=code&client_id=lh_platform&redirect_uri=http:%2F%2Fucenter.sdland-sea.com%2F%23%2FcodeCallback%3Fredirect_uri%3Dhttp%253A%252F%252Fucenter.sdland-sea.com%252F%2523%252F  223.104.189.25 
    [19/May/2020:11:00:30 /static/js/4.a41a20d9223eaaa4eb4c.js  223.104.189.25 
    [19/May/2020:11:00:29 /static/js/vendor.e4ef0e9a82c50cc8468b.js  223.104.189.25 
    [19/May/2020:11:00:29 /static/js/manifest.5f19f0f424e5074bc361.js  223.104.189.25 
    [19/May/2020:11:00:29 /static/js/app.5820bd1c4e778787d807.js  223.104.189.25 
    [19/May/2020:11:00:29 /api/lh-xxwh/certification/getUserMenu  223.104.189.25