Vert.x memory usage with intensive network
I recently worked on Vert.x memory issues and I would like to share some of the knowledge I gained. Vert.x memory usage can be tricky especially when doing intensive networking. Java developers are usually concerned only about Java heap memory, but in Vert.x it is different.
Vert.x memory usage
Vert.x is reactive scaleable system with impressive capabilities. Vert.x allows fast communication to many clients simultaneously, but you may not notice that Vert.x depends heavily on Netty for networking. Netty is high performance network library. But how does Netty do things so fast ? One of the reasons is that it is using native memory pool to store network buffers. If you did some file reading or network action with Vert.x you probably used io.vertx.core.buffer.Buffer class. This class is actually a wrapper around Netty io.netty.buffer.ByteBuf class. Why am I telling you all this ? Let assume that you have a service where clients are downloading 20Mbyte files. Netty will have to allocate at least 20Mbyte for every connected client. Now if there are 100 client connected simultaneously then Netty will need at list 2Gbyte of available memory. If the server does not have 2Gbyte of available memory the application may crash or be very slow. When I had a Vert.x application crashing I had no idea what is consuming so much memory. I thought I had a native memory leak and after long hours of profiling I found that Netty is consuming a lot of memory. The first thing I did was trying to limit Netty memory, assuming that Netty has memory leak. I ran the application with the following argument:
But in my case this only made thing worse, Netty was out of memory and was returning null buffers which eventually got sent to the client and caused the client to crash - ouch !!! Vert.x did not limit the number of connected client, the only limitation was the server resources. I then realised that I have too many clients connected simultaneously and I need to limit the number of connected clients.
When using Vert.x you have to consider that you are also using Netty, you need to have enough available memory for Netty to use. You may also need to implement some king of throttling mechanism to limit the number of clients being handled.