Netty 100万级高并发服务器配置

朱雀 2023-05-29 05:07 233阅读 0赞

**疯狂创客圈，一个Java 高并发研习社群** 【[博客园 总入口][Link 1] 】

疯狂创客圈，倾力推出：《[Netty Zookeeper Redis 高并发实战][Netty Zookeeper Redis]》 **面试必备 + 面试必备 + 面试必备** 的 基础原理+实战 书籍

![书籍][format_png]

--------------------

## 前言 ##

每一种该语言在某些极限情况下的表现一般都不太一样，那么我常用的Java语言，在达到100万个并发连接情况下，会怎么样呢，有些好奇，更有些期盼。  
这次使用经常使用的顺手的**netty** NIO框架（netty-3.6.5.Final），封装的很好，接口很全面，就像它现在的域名 **netty.io**，专注于网络IO。  
整个过程没有什么技术含量，浅显分析过就更显得有些枯燥无聊，准备好，硬着头皮吧。

## 测试服务器配置 ##

运行在VMWare Workstation 9中，64位Centos 6.2系统，分配14.9G内存左右，4核。  
已安装有Java7版本：

java version "1.7.0_21"
    Java(TM) SE Runtime Environment (build 1.7.0_21-b11)
    Java HotSpot(TM) 64-Bit Server VM (build 23.21-b01, mixed mode)
    
      
       
      
       
        1
        2
        3

在/etc/sysctl.conf中添加如下配置：

fs.file-max = 1048576
    net.ipv4.ip_local_port_range = 1024 65535
    net.ipv4.tcp_mem = 786432 2097152 3145728
    net.ipv4.tcp_rmem = 4096 4096 16777216
    net.ipv4.tcp_wmem = 4096 4096 16777216

net.ipv4.tcp\_tw\_reuse = 1  
net.ipv4.tcp\_tw\_recycle = 1

*  1
 *  2
 *  3
 *  4
 *  5
 *  6
 *  7
 *  8

在/etc/security/limits.conf中添加如下配置：

*	soft nofile 1048576
         *	hard nofile 1048576
    
     
      
     
      
       1
       2

## 测试端 ##

测试端无论是配置还是程序和以前一样，翻看前几篇博客就可以看到client5.c的源码，以及相关的配置信息等。

## 服务器程序 ##

这次也是很简单呐，没有业务功能，客户端HTTP请求，服务端输出chunked编码内容。

### 入口HttpChunkedServer.java： ###

package com.test.server;

import static org.jboss.netty.channel.Channels.pipeline;  
import java.net.InetSocketAddress;  
import java.util.concurrent.Executors;  
import org.jboss.netty.bootstrap.ServerBootstrap;  
import org.jboss.netty.channel.ChannelPipeline;  
import org.jboss.netty.channel.ChannelPipelineFactory;  
import org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory;  
import org.jboss.netty.handler.codec.http.HttpChunkAggregator;  
import org.jboss.netty.handler.codec.http.HttpRequestDecoder;  
import org.jboss.netty.handler.codec.http.HttpResponseEncoder;  
import org.jboss.netty.handler.stream.ChunkedWriteHandler;

public class HttpChunkedServer \{

private final int port;
    
    public HttpChunkedServer(intport) {
        this.port = port;
    }
    
    public void run() {
        // Configure the server.
        ServerBootstrap bootstrap = new ServerBootstrap(
                new NioServerSocketChannelFactory(
                        Executors.newCachedThreadPool(),
                        Executors.newCachedThreadPool()));
    
        // Set up the event pipeline factory.
        bootstrap.setPipelineFactory(newChannelPipelineFactory() {
            public ChannelPipeline getPipeline ()throws Exception {
                ChannelPipeline pipeline = pipeline();
                pipeline.addLast("decoder", new HttpRequestDecoder());
                pipeline.addLast("aggregator", new HttpChunkAggregator(65536));
                pipeline.addLast("encoder", new HttpResponseEncoder());
                pipeline.addLast("chunkedWriter", new ChunkedWriteHandler());
                pipeline.addLast("handler", new HttpChunkedServerHandler());
                return pipeline;
            }
        });
    
        bootstrap.setOption("child.reuseAddress", true);
        bootstrap.setOption("child.tcpNoDelay", true);
        bootstrap.setOption("child.keepAlive", true);
        // Bind and start to accept incoming connections.
        bootstrap.bind(newInetSocketAddress(port));
    }
    
    public static void main(String[] args) {
        int port;
        if (args.length > 0) {
            port = Integer.parseInt(args[0]);
        } else {
            port = 8080;
        }
    
        System.out.format("server start with port %d \n", port);
        new HttpChunkedServer(port).run();
    }

### 唯一的自定义处理器HttpChunkedServerHandler.java: ###

package com.test.server;

import static org.jboss.netty.handler.codec.http.HttpHeaders.Names.CONTENT\_TYPE;  
import static org.jboss.netty.handler.codec.http.HttpMethod.GET;  
import static org.jboss.netty.handler.codec.http.HttpResponseStatus.BAD\_REQUEST;  
import static org.jboss.netty.handler.codec.http.HttpResponseStatus.METHOD\_NOT\_ALLOWED;  
import static org.jboss.netty.handler.codec.http.HttpResponseStatus.OK;  
import static org.jboss.netty.handler.codec.http.HttpVersion.HTTP\_1\_1;  
import java.util.concurrent.atomic.AtomicInteger;  
import org.jboss.netty.buffer.ChannelBuffer;  
import org.jboss.netty.buffer.ChannelBuffers;  
import org.jboss.netty.channel.Channel;  
import org.jboss.netty.channel.ChannelFutureListener;  
import org.jboss.netty.channel.ChannelHandlerContext;  
import org.jboss.netty.channel.ChannelStateEvent;  
import org.jboss.netty.channel.ExceptionEvent;  
import org.jboss.netty.channel.MessageEvent;  
import org.jboss.netty.channel.SimpleChannelUpstreamHandler;  
import org.jboss.netty.handler.codec.frame.TooLongFrameException;  
import org.jboss.netty.handler.codec.http.DefaultHttpChunk;  
import org.jboss.netty.handler.codec.http.DefaultHttpResponse;  
import org.jboss.netty.handler.codec.http.HttpChunk;  
import org.jboss.netty.handler.codec.http.HttpHeaders;  
import org.jboss.netty.handler.codec.http.HttpRequest;  
import org.jboss.netty.handler.codec.http.HttpResponse;  
import org.jboss.netty.handler.codec.http.HttpResponseStatus;  
import org.jboss.netty.util.CharsetUtil;

public class HttpChunkedServerHandlerextends SimpleChannelUpstreamHandler \{  
private static final AtomicInteger count = new AtomicInteger(0);

private void increment() {
        System.out.format("online user %d\n", count.incrementAndGet());
    }
    
    private void decrement() {
        if (count.get() <= 0) {
            System.out.format("~online user %d\n", 0);
        } else {
            System.out.format("~online user %d\n", count.decrementAndGet());
        }
    }
    
    @Override
    public void messageReceived(ChannelHandlerContextctx, MessageEvent e)
            throws Exception {
        HttpRequest request = (HttpRequest) e.getMessage();
        if (request.getMethod() != GET) {
            sendError(ctx, METHOD_NOT_ALLOWED);
            return;
        }
        
        sendPrepare(ctx);
        increment();
    }
    
    @Override
    public void channelDisconnected(ChannelHandlerContextctx,
                                    ChannelStateEvent e) throws Exception {
        decrement();
        super.channelDisconnected(ctx, e);
    }
    
    @Override
    public void exceptionCaught(ChannelHandlerContextctx, ExceptionEvent e)
            throws Exception {
        Throwable cause = e.getCause();
        if (cause instanceof TooLongFrameException) {
            sendError(ctx, BAD_REQUEST);
            return;
        }
    }
    
    private static void sendError(ChannelHandlerContext ctx,
                                  HttpResponseStatus status) {
        HttpResponse response = new DefaultHttpResponse(HTTP_1_1, status);
        response.setHeader(CONTENT_TYPE, "text/plain; charset=UTF-8");
        response.setContent(ChannelBuffers.copiedBuffer(
                "Failure:" + status.toString() + "\r\n", CharsetUtil.UTF_8));
        
        // Close the connection as soon as the error message is sent.
        ctx.getChannel().write(response)
                .addListener(ChannelFutureListener.CLOSE);
    }
    
    private void sendPrepare(ChannelHandlerContextctx) {
        HttpResponse response = new DefaultHttpResponse(HTTP_1_1, OK);
        response.setChunked(true);
        response.setHeader(HttpHeaders.Names.CONTENT_TYPE,
                "text/html; charset=UTF-8");
        response.addHeader(HttpHeaders.Names.CONNECTION,
                HttpHeaders.Values.KEEP_ALIVE);
        response.setHeader(HttpHeaders.Names.TRANSFER_ENCODING,
                HttpHeaders.Values.CHUNKED);
        
        Channel chan = ctx.getChannel();
        chan.write(response);
        
        // 缓冲必须凑够256字节，浏览器端才能够正常接收 ...
        StringBuilder builder = new StringBuilder();
        builder.append("");
        int leftChars = 256 - builder.length();
        for (int i = 0; i < leftChars; i++) {
            builder.append("");
        }
        
        writeStringChunk(chan, builder.toString());
    }
    
    private void writeStringChunk(Channelchannel, String data) {
        ChannelBuffer chunkContent = ChannelBuffers.dynamicBuffer(channel
                .getConfig().getBufferFactory());
        chunkContent.writeBytes(data.getBytes());
        HttpChunk chunk = new DefaultHttpChunk(chunkContent);
        channel.write(chunk);
    }

*  1
 *  2
 *  3
 *  4
 *  5
 *  6
 *  7
 *  8
 *  9
 *  10
 *  11
 *  12
 *  13
 *  14
 *  15
 *  16
 *  17
 *  18
 *  19
 *  20
 *  21
 *  22
 *  23
 *  24
 *  25
 *  26
 *  27
 *  28
 *  29
 *  30
 *  31
 *  32
 *  33
 *  34
 *  35
 *  36
 *  37
 *  38
 *  39
 *  40
 *  41
 *  42
 *  43
 *  44
 *  45
 *  46
 *  47
 *  48
 *  49
 *  50
 *  51
 *  52
 *  53
 *  54
 *  55
 *  56
 *  57
 *  58
 *  59
 *  60
 *  61
 *  62
 *  63
 *  64
 *  65
 *  66
 *  67
 *  68
 *  69
 *  70
 *  71
 *  72
 *  73
 *  74
 *  75
 *  76
 *  77
 *  78
 *  79
 *  80
 *  81
 *  82
 *  83
 *  84
 *  85
 *  86
 *  87
 *  88
 *  89
 *  90
 *  91
 *  92
 *  93
 *  94
 *  95
 *  96
 *  97
 *  98
 *  99
 *  100
 *  101
 *  102
 *  103
 *  104
 *  105
 *  106
 *  107
 *  108
 *  109
 *  110
 *  111
 *  112
 *  113
 *  114
 *  115
 *  116
 *  117

### 启动脚本start.sh ###

set CLASSPATH=.
    nohup java -server -Xmx6G -Xms6G -Xmn600M -XX:PermSize=50M -XX:MaxPermSize=50M -Xss256K -XX:+DisableExplicitGC -XX:SurvivorRatio=1 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSParallelRemarkEnabled -XX:+UseCMSCompactAtFullCollection -XX:CMSFullGCsBeforeCompaction=0 -XX:+CMSClassUnloadingEnabled -XX:LargePageSizeInBytes=128M -XX:+UseFastAccessorMethods -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=80 -XX:SoftRefLRUPolicyMSPerMB=0 -XX:+PrintClassHistogram -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -Xloggc:gc.log -Djava.ext.dirs=lib com.test.server.HttpChunkedServer 8000>server.out 2>&1 &
    
     
      
     
      
       1
       2

## 达到100万并发连接时的一些信息 ##

每次服务器端达到一百万个并发持久连接之后，然后关掉测试端程序，断开所有的连接，等到服务器端日志输出在线用户为0时，再次重复以上步骤。在这反反复复的情况下，观察内存等信息的一些情况。以某次断开所有测试端为例后，当前系统占用为（设置为`list_free_1`）:

total       used       free     shared    buffers     cached
         Mem:         15189       7736       7453          0         18        120
         -/+ buffers/cache:       7597       7592
         Swap:         4095        948       3147
    
     
      
     
      
       1
       2
       3
       4

通过top观察，其进程相关信息

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                       
       4925 root      20   0 8206m 4.3g 2776 S  0.3 28.8  50:18.66 java
    
     
      
     
      
       1
       2

在启动脚本start.sh中，我们设置堆内存为6G。

**ps aux|grep java**命令获得信息：

root      4925 38.0 28.8 8403444 4484764 ?     Sl   15:26  50:18 java -server...HttpChunkedServer 8000
    
     
      
     
      
       1

RSS占用内存为4484764K/1024K=4379M

然后再次启动测试端，在服务器接收到**online user 1023749**时，`ps aux|grep java`内容为：

root      4925 43.6 28.4 8403444 4422824 ?     Sl   15:26  62:53 java -server...
    
     
      
     
      
       1

查看当前网络信息统计

ss -s
      Total: 1024050 (kernel 1024084)
      TCP:   1023769 (estab 1023754, closed 2, orphaned 0, synrecv 0, timewait 0/0), ports 12

Transport Total IP IPv6

*  1024084 - -  
    RAW 0 0 0  
    UDP 7 6 1  
    TCP 1023767 12 1023755  
    INET 1023774 18 1023756  
    FRAG 0 0 0

*  1
 *  2
 *  3
 *  4
 *  5
 *  6
 *  7
 *  8
 *  9
 *  10
 *  11

通过top查看一下

top -p 4925
      top - 17:51:30 up  3:02,  4 users,  load average: 1.03, 1.80, 1.19
      Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
      Cpu0  :  0.9%us,  2.6%sy,  0.0%ni, 52.9%id,  1.0%wa, 13.6%hi, 29.0%si,  0.0%st
      Cpu1  :  1.4%us,  4.5%sy,  0.0%ni, 80.1%id,  1.9%wa,  0.0%hi, 12.0%si,  0.0%st
      Cpu2  :  1.5%us,  4.4%sy,  0.0%ni, 80.5%id,  4.3%wa,  0.0%hi,  9.3%si,  0.0%st
      Cpu3  :  1.9%us,  4.4%sy,  0.0%ni, 84.4%id,  3.2%wa,  0.0%hi,  6.2%si,  0.0%st
      Mem:  15554336k total, 15268728k used,   285608k free,     3904k buffers
      Swap:  4194296k total,  1082592k used,  3111704k free,    37968k cached
     PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND

4925 root 20 0 8206m 4.2g 2220 S 3.3 28.4 62:53.66 java

*  1
 *  2
 *  3
 *  4
 *  5
 *  6
 *  7
 *  8
 *  9
 *  10
 *  11
 *  12

四核都被占用了，每一个核心不太平均。这是在虚拟机中得到结果，可能真实服务器会更好一些。 因为不是CPU密集型应用，CPU不是问题，无须多加关注。

系统内存状况

free -m
                   total       used       free     shared    buffers     cached
      Mem:         15189      14926        263          0          5         56
      -/+ buffers/cache:      14864        324
      Swap:         4095       1057       3038
    
     
      
     
      
       1
       2
       3
       4
       5

物理内存已经无法满足要求了，占用了1057M虚拟内存。

查看一下堆内存情况

jmap -heap 4925
      Attaching to process ID 4925, please wait...
      Debugger attached successfully.
      Server compiler detected.
      JVM version is 23.21-b01

using parallel threads in the new generation.  
using thread-local object allocation.  
Concurrent Mark-Sweep GC

Heap Configuration:  
MinHeapFreeRatio = 40  
MaxHeapFreeRatio = 70  
MaxHeapSize = 6442450944 (6144.0MB)  
NewSize = 629145600 (600.0MB)  
MaxNewSize = 629145600 (600.0MB)  
OldSize = 5439488 (5.1875MB)  
NewRatio = 2  
SurvivorRatio = 1  
PermSize = 52428800 (50.0MB)  
MaxPermSize = 52428800 (50.0MB)  
G1HeapRegionSize = 0 (0.0MB)

Heap Usage:  
New Generation (Eden + 1 Survivor Space):  
capacity = 419430400 (400.0MB)  
used = 308798864 (294.49354553222656MB)  
free = 110631536 (105.50645446777344MB)  
73.62338638305664% used  
Eden Space:  
capacity = 209715200 (200.0MB)  
used = 103375232 (98.5863037109375MB)  
free = 106339968 (101.4136962890625MB)  
49.29315185546875% used  
From Space:  
capacity = 209715200 (200.0MB)  
used = 205423632 (195.90724182128906MB)  
free = 4291568 (4.0927581787109375MB)  
97.95362091064453% used  
To Space:  
capacity = 209715200 (200.0MB)  
used = 0 (0.0MB)  
free = 209715200 (200.0MB)  
0.0% used  
concurrent mark-sweep generation:  
capacity = 5813305344 (5544.0MB)  
used = 4213515472 (4018.321487426758MB)  
free = 1599789872 (1525.6785125732422MB)  
72.48054631000646% used  
Perm Generation:  
capacity = 52428800 (50.0MB)  
used = 5505696 (5.250640869140625MB)  
free = 46923104 (44.749359130859375MB)  
10.50128173828125% used

1439 interned Strings occupying 110936 bytes.

老生代占用内存为72%，较为合理，毕竟系统已经处理100万个连接。

再次断开所有测试端，看看系统内存（free -m）

total       used       free     shared    buffers     cached
      Mem:         15189       7723       7466          0         13        120
      -/+ buffers/cache:       7589       7599
      Swap:         4095        950       3145
    
     
      
     
      
       1
       2
       3
       4

记为`list_free_2`。

`list_free_1`和`list_free_2`两次都释放后的内存比较结果，系统可用物理已经内存已经降到7589M，先前可是7597M物理内存。  
总之，我们的JAVA测试程序在内存占用方面已经，最低需要7589 + 950 = 8.6G内存为最低需求内存吧。

## GC日志 ##

我们在启动脚本处设置的一大串参数，到底是否达到目标，还得从gc日志处获得具体效果，推荐使用[GCViewer][]。

GC事件概览：  
[![gc\_eventdetails][gc_eventdetails]][gc_eventdetails_gc_eventdetails]

其它:  
[![gc\_total\_1][gc_total_1]][gc_total_1_gc_total_1] [![gc\_total\_2][gc_total_2]][gc_total_2_gc_total_2] [![gc\_total\_3][gc_total_3]][gc_total_3_gc_total_3]

总之：

*  只进行了一次Full GC，代价太高，停顿了12秒。
 *  PartNew成为了停顿大户，导致整个系统停顿了41秒之久，不可接受。
 *  当前JVM调优喜忧参半，还得继续努力等

## 小结 ##

Java与与Erlang、C相比，比较麻烦的事情，需要在程序一开始就得准备好它的堆栈到底需要多大空间，换个说法就是JVM启动参数设置堆内存大小，设置合适的垃圾回收机制，若以后程序需要更多内存，需停止程序，编辑启动参数，然后再次启动。总之一句话，就是麻烦。单单JVM的调优，就得持续不断的根据检测、信息、日志等进行适当微调。

*  JVM需要提前指定堆大小，相比Erlang/C，这可能是个麻烦
 *  GC(垃圾回收)，相对比麻烦，需要持续不断的根据日志、JVM堆栈信息、运行时情况进行JVM参数微调
 *  设置一个最大连接目标，多次测试达到顶峰，然后释放所有连接，反复观察内存占用，获得一个较为合适的系统运行内存值
 *  Eclipse Memory Analyzer结合jmap导出堆栈DUMP文件，分析内存泄漏，还是很方便的
 *  想修改运行时内容，或者称之为热加载，默认不可能
 *  真实机器上会有更好的反映

吐槽一下：  
JAVA OSGI，相对比Erlang来说，需要人转换思路，不是那么原生的东西，总是有些别扭，社区或商业公司对此的修修补补，不过是实现一些面向对象所不具备的热加载的企业特性。

测试源代码，[下载just\_test][just_test]。

--------------------

**无编程不创客，无案例不学习。疯狂创客圈，一大波高手正在交流、学习中!**

**疯狂创客圈 Netty 死磕系列 10多篇深度文章： 【**[**博客园 总入口**][Link 1]**】 QQ群：104131248**

</div>

[Link 1]: https://www.cnblogs.com/crazymakercircle/p/9904544.html
[Netty Zookeeper Redis]: https://www.cnblogs.com/crazymakercircle/p/11397271.html
[format_png]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly9pbWcyMDE4LmNuYmxvZ3MuY29tL2Jsb2cvMTQ4NTM5OC8yMDE5MDgvMTQ4NTM5OC0yMDE5MDgyMjIyNTE1NjQyNy05NTY0MjQxMjQuanBn?x-oss-process=image/format,png
[GCViewer]: https://github.com/chewiebug/GCViewer
[gc_eventdetails]: https://imgconvert.csdnimg.cn/aHR0cDovL3d3dy5ibG9namF2YS5uZXQvaW1hZ2VzL2Jsb2dqYXZhX25ldC95b25nYm95L1dpbmRvd3MtTGl2ZS1Xcml0ZXIvYmYzZWI1YWQ5ZGQ5XzlDN0YvZ2NfZXZlbnRkZXRhaWxzX3RodW1iLnBuZw?x-oss-process=image/format,png
[gc_eventdetails_gc_eventdetails]: http://www.blogjava.net/images/blogjava_net/yongboy/Windows-Live-Writer/bf3eb5ad9dd9_9C7F/gc_eventdetails_2.png
[gc_total_1]: https://imgconvert.csdnimg.cn/aHR0cDovL3d3dy5ibG9namF2YS5uZXQvaW1hZ2VzL2Jsb2dqYXZhX25ldC95b25nYm95L1dpbmRvd3MtTGl2ZS1Xcml0ZXIvYmYzZWI1YWQ5ZGQ5XzlDN0YvZ2NfdG90YWxfMV90aHVtYi5wbmc?x-oss-process=image/format,png
[gc_total_1_gc_total_1]: http://www.blogjava.net/images/blogjava_net/yongboy/Windows-Live-Writer/bf3eb5ad9dd9_9C7F/gc_total_1_2.png
[gc_total_2]: https://imgconvert.csdnimg.cn/aHR0cDovL3d3dy5ibG9namF2YS5uZXQvaW1hZ2VzL2Jsb2dqYXZhX25ldC95b25nYm95L1dpbmRvd3MtTGl2ZS1Xcml0ZXIvYmYzZWI1YWQ5ZGQ5XzlDN0YvZ2NfdG90YWxfMl90aHVtYi5wbmc?x-oss-process=image/format,png
[gc_total_2_gc_total_2]: http://www.blogjava.net/images/blogjava_net/yongboy/Windows-Live-Writer/bf3eb5ad9dd9_9C7F/gc_total_2_2.png
[gc_total_3]: https://imgconvert.csdnimg.cn/aHR0cDovL3d3dy5ibG9namF2YS5uZXQvaW1hZ2VzL2Jsb2dqYXZhX25ldC95b25nYm95L1dpbmRvd3MtTGl2ZS1Xcml0ZXIvYmYzZWI1YWQ5ZGQ5XzlDN0YvZ2NfdG90YWxfM190aHVtYi5wbmc?x-oss-process=image/format,png
[gc_total_3_gc_total_3]: http://www.blogjava.net/images/blogjava_net/yongboy/Windows-Live-Writer/bf3eb5ad9dd9_9C7F/gc_total_3_2.png
[just_test]: http://www.blogjava.net/Files/yongboy/just_test.zip