Nginx日志分析统计 goaccess-1.3部署与使用

迈不过友情╰ 2023-02-20 10:58 5阅读 0赞

其实别人已经写得很好了，但是我将几个需要注意的点给标注一下。

需求：及时得到线上用户nginx访问日志分析统计结果！我需要知道IP访问排行、链接访问排行、静态资源、404错误排行等  
直接上效果图：

![watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lleGlhb21vZGVtbw_size_16_color_FFFFFF_t_70][]

### 具体安装步骤如下：   1、安装依赖 ###

[root@iZbp1f0xuq9rc41s6gdvfyZ ~]# mkdir access
    [root@iZbp1f0xuq9rc41s6gdvfyZ access]# yum install glib2 glib2-devel GeoIP-devel  ncurses-devel zlib zlib-devel -y
    [root@iZbp1f0xuq9rc41s6gdvfyZ access]# wget https://github.com/maxmind/geoip-api-c/releases/download/v1.6.11/GeoIP-1.6.11.tar.gz
    [root@iZbp1f0xuq9rc41s6gdvfyZ access]# tar -zvxf GeoIP-1.6.11.tar.gz
    [root@iZbp1f0xuq9rc41s6gdvfyZ access]# cd GeoIP-1.6.11
    [root@iZbp1f0xuq9rc41s6gdvfyZ GeoIP-1.6.11]# ./configure
    [root@iZbp1f0xuq9rc41s6gdvfyZ GeoIP-1.6.11]# make && make install

### 2、安装goaccess ###

[root@iZbp1f0xuq9rc41s6gdvfyZ GeoIP-1.6.11]# cd ..
    [root@iZbp1f0xuq9rc41s6gdvfyZ access]# wget https://tar.goaccess.io/goaccess-1.3.tar.gz
    [root@iZbp1f0xuq9rc41s6gdvfyZ access]# tar -xzvf goaccess-1.3.tar.gz
    [root@iZbp1f0xuq9rc41s6gdvfyZ access]# cd goaccess-1.3
    [root@iZbp1f0xuq9rc41s6gdvfyZ goaccess-1.3]# ./configure --enable-utf8 --enable-geoip=legacy
    [root@iZbp1f0xuq9rc41s6gdvfyZ goaccess-1.3]# make && make install

### 3、校对nginx和goaccess日志格式   查看当前的nginx配置，日志格式 ###

[root@iZbp1f0xuq9rc41s6gdvfyZ conf]# pwd
    /usr/local/nginx/conf
    [root@iZbp1f0xuq9rc41s6gdvfyZ conf]# cat /usr/local/nginx/conf/nginx.conf

![2020062616311594.png][]

记住这一段，后面有用。每个机器配置可能不一样哦，使用你自己的

log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
    		'$status $body_bytes_sent "$http_referer" ' 
    		'"$http_user_agent" "$http_x_forwarded_for" "$request_body"';

### 4、goaccess日志格式配置 ###

一：使用nginx2goaccess.sh脚本将nginx日志格式格式化为goaccess能识别的日志格式，nginx2goaccess.sh脚本内容在

[root@iZbp1f0xuq9rc41s6gdvfyZ goaccess-1.3]# vi nginx2goaccess.sh
    
    #!/bin/bash
    #
    # Convert from this:
    #   http://nginx.org/en/docs/http/ngx_http_log_module.html
    # To this:
    #   https://goaccess.io/man
    #
    # Conversion table:
    #   $time_local         %d:%t %^
    #   $host               %v
    #   $http_host          %v
    #   $remote_addr        %h
    #   $request_time       %T
    #   $request_method     %m
    #   $request_uri        %U
    #   $server_protocol    %H
    #   $request            %r
    #   $status             %s
    #   $body_bytes_sent    %b
    #   $bytes_sent         %b
    #   $http_referer       %R
    #   $http_user_agent    %u
    #
    # Samples:
    #
    # log_format combined '$remote_addr - $remote_user [$time_local] '
    # '"$request" $status $body_bytes_sent '
    # '"$http_referer" "$http_user_agent"';
    #   ./nginx2goaccess.sh '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"'
    #
    # log_format compression '$remote_addr - $remote_user [$time_local] '
    # '"$request" $status $bytes_sent '
    # '"$http_referer" "$http_user_agent" "$gzip_ratio"';
    #   ./nginx2goaccess.sh '$remote_addr - $remote_user [$time_local] "$request" $status $bytes_sent "$http_referer" "$http_user_agent" "$gzip_ratio"'
    #
    # log_format main
    # '$remote_addr\t$time_local\t$host\t$request\t$http_referer\t$http_x_mobile_group\t'
    # 'Local:\t$status\t$body_bytes_sent\t$request_time\t'
    # 'Proxy:\t$upstream_cache_status\t$upstream_status\t$upstream_response_length\t$upstream_response_time\t'
    # 'Agent:\t$http_user_agent\t'
    # 'Fwd:\t$http_x_forwarded_for';
    #   ./nginx2goaccess.sh '$remote_addr\t$time_local\t$host\t$request\t$http_referer\t$http_x_mobile_group\tLocal:\t$status\t$body_bytes_sent\t$request_time\tProxy:\t$upstream_cache_status\t$upstream_status\t$upstream_response_length\t$upstream_response_time\tAgent:\t$http_user_agent\tFwd:\t$http_x_forwarded_for'
    #
    # log_format main
    # '${time_local}\t${remote_addr}\t${host}\t${request_method}\t${request_uri}\t${server_protocol}\t'
    # '${http_referer}\t${http_x_mobile_group}\t'
    # 'Local:\t${status}\t*${connection}\t${body_bytes_sent}\t${request_time}\t'
    # 'Proxy:\t${upstream_status}\t${upstream_cache_status}\t'
    # '${upstream_response_length}\t${upstream_response_time}\t${uri}${log_args}\t'
    # 'Agent:\t${http_user_agent}\t'
    # 'Fwd:\t${http_x_forwarded_for}';
    #   ./nginx2goaccess.sh '${time_local}\t${remote_addr}\t${host}\t${request_method}\t${request_uri}\t${server_protocol}\t${http_referer}\t${http_x_mobile_group}\tLocal:\t${status}\t*${connection}\t${body_bytes_sent}\t${request_time}\tProxy:\t${upstream_status}\t${upstream_cache_status}\t${upstream_response_length}\t${upstream_response_time}\t${uri}${log_args}\tAgent:\t${http_user_agent}\tFwd:\t${http_x_forwarded_for}'
    #
    # Author: Rogério Carvalho Schneider <stockrt@gmail.com>
    
    # Params
    log_format="$1"
    
    # Usage
    if [[ -z "$log_format" ]]; then
        echo "Usage: $0 '<log_format>'"
        exit 1
    fi
    
    # Variables map
    conversion_table="time_local,%d:%t_%^
    host,%v
    http_host,%v
    remote_addr,%h
    request_time,%T
    request_method,%m
    request_uri,%U
    server_protocol,%H
    request,%r
    status,%s
    body_bytes_sent,%b
    bytes_sent,%b
    http_referer,%R
    http_user_agent,%u"
    
    # Conversion
    for item in $conversion_table; do
        nginx_var=${item%%,*}
        goaccess_var=${item##*,}
        goaccess_var=${goaccess_var//_/ }
        log_format=${log_format//\$\{$nginx_var\}/$goaccess_var}
        log_format=${log_format//\$$nginx_var/$goaccess_var}
    done
    log_format=$(echo "$log_format" | sed 's/${[a-z_]*}/%^/g')
    log_format=$(echo "$log_format" | sed 's/$[a-z_]*/%^/g')
    
    # Config output
    echo "
    - Generated goaccess config:
    time-format %T
    date-format %d/%b/%Y
    log_format $log_format
    "
    
    # EOF

二：使用nginx2goaccess.sh方式获取日志格式

将第二步里面Nginx日志的单引号删除：
    
    删除前：'$remote_addr - $remote_user [$time_local] "$request" '
    		'$status $body_bytes_sent "$http_referer" ' 
    		'"$http_user_agent" "$http_x_forwarded_for" "$request_body"'
    
    删除后：$remote_addr - $remote_user [$time_local] "$request" 
    		$status $body_bytes_sent "$http_referer" 
    		"$http_user_agent" "$http_x_forwarded_for" "$request_body"
    
    然后执行命令如：
    [root@iZbp1f0xuq9rc41s6gdvfyZ goaccess-1.3]# sh nginx2goaccess.sh '删除后'
    执行真实命令：
    [root@iZbp1f0xuq9rc41s6gdvfyZ goaccess-1.3]# sh nginx2goaccess.sh '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" "$http_x_forwarded_for" "$request_body"'

得到如下结果：

- Generated goaccess config:
    time-format %T
    date-format %d/%b/%Y
    log_format %h - %^ [%d:%t %^] "%r" %s %b "%R" "%u" "%^" "%r_body"

三：将上面生成的格式在goaccess-1.3/config下面创建一个nginxlog.conf

[root@iZbp1f0xuq9rc41s6gdvfyZ goaccess-1.3]# cd config
    
    [root@iZbp1f0xuq9rc41s6gdvfyZ config]# vi nginxlog.conf
    
    # Generated goaccess config:
    time-format %T
    date-format %d/%b/%Y
    log_format %h - %^ [%d:%t %^] "%r" %s %b "%R" "%u" "%^" "%r_body"

### 5、配置完成，开始见证奇迹的时刻，生成统计页面。 ###

第一步：生成结果文件夹的位置   /data/nginx
    [root@iZbp1f0xuq9rc41s6gdvfyZ /]# mkdir /data
    [root@iZbp1f0xuq9rc41s6gdvfyZ /]# cd data/
    [root@iZbp1f0xuq9rc41s6gdvfyZ data]# mkdir nginx
    
    第二步：nginx日志位置、最好做了每日切割，否则太大了
    [root@iZbp1f0xuq9rc41s6gdvfyZ logs]# pwd
    /usr/local/nginx/logs
    
    第三步：开始日志分析，需要回到goaccess-1.3 文件夹
    [root@iZbp1f0xuq9rc41s6gdvfyZ goaccess-1.3]# ./goaccess -f /usr/local/nginx/logs/access.log -p config/nginxlog.conf -o /data/nginx/report.html
    
    第四步：生成中文的方法
    [root@iZbp1f0xuq9rc41s6gdvfyZ goaccess-1.3]# LANG="zh_CN.UTF-8" bash -c "./goaccess -f /usr/local/nginx/logs/access.log -p config/nginxlog.conf -o /data/nginx/report.html"
    
    说明：第三步、第四步分别生成的是当前时间的日志分析结果，五分钟之后想看新的得再执行一次
    下面我们写个定时器，让五分钟分析一次
    
    下面配置定时更新
    第五步：创建脚本
    [root@iZbp1f0xuq9rc41s6gdvfyZ goaccess-1.3]# mkdir scripts
    [root@iZbp1f0xuq9rc41s6gdvfyZ goaccess-1.3]# cd scripts
    [root@iZbp1f0xuq9rc41s6gdvfyZ scripts]# vi goaccess.sh
    
    #!/bin/bash
    LANG="zh_CN.UTF-8"           #页面转换为中文
    "/root/goaccess-1.3/goaccess -f /usr/local/nginx/logs/access.log -p /root/goaccess-1.3/config/nginxlog.conf -o /data/nginx/report.html"
    
    第六步：加入定时器
    [root@iZbp1f0xuq9rc41s6gdvfyZ scripts]# chmod + ./goaccess.sh
    [root@iZbp1f0xuq9rc41s6gdvfyZ scripts]# pwd
    /root/goaccess-1.3/scripts
    [root@iZbp1f0xuq9rc41s6gdvfyZ scripts]# crontab -e
    # 定时goaccess 每五分钟生成的信息保存为html
    5 * * * * /root/goaccess-1.3/scripts/goaccess.sh

### 6、配置nginx的配置文件，可访问 ###

配置nginx访问刚刚生成的report.html页面，阿里云的朋友自行配置即可，没多大的区别！

[root@iZbp1f0xuq9rc41s6gdvfyZ conf]# pwd
    /usr/local/nginx/conf
    
    [root@iZbp1f0xuq9rc41s6gdvfyZ conf]# vi nginx.conf
    加上此段：
    
    server{
    	listen 8080;
    	server_name localhost;
    
    	location /report.html {
                alias /data/nginx/report.html;
            } 
    }

重启nginx

[root@iZbp1f0xuq9rc41s6gdvfyZ conf]# /usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/nginx.conf

在浏览器就 [http://192.168.2.253:8080/report.html][http_192.168.2.253_8080_report.html] 可以浏览上上面的截图了。

[watermark_type_ZmFuZ3poZW5naGVpdGk_shadow_10_text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lleGlhb21vZGVtbw_size_16_color_FFFFFF_t_70]: https://img-blog.csdnimg.cn/20200626171213566.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3lleGlhb21vZGVtbw==,size_16,color_FFFFFF,t_70
[2020062616311594.png]: https://img-blog.csdnimg.cn/2020062616311594.png
[http_192.168.2.253_8080_report.html]: http://192.168.1.50:8080/report.html