Varnish Using Significantly More Memory than Cache?

Question

I'm using varnish as a web cache. I'm running on RHEL 9.2. My cache is sized at 1GB. My varnish process is using 3.7G memory.

$ ps -p 1163  -o %mem,rss
%MEM   RSS
15.7 3886108

I'm running varnish with the -s malloc,1024M argument, which should set the cache size to 1GB. The full command, being invoked by systemd, is:

/usr/sbin/varnishd -a 127.6.6.6:7480 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s malloc,1024M -P /var/run/varnish.pid -T 127.6.6.6:62

It appears that is working, I can see with varnishstat:

$ varnishstat -1 | grep s0.g
SMA.s0.g_alloc                            22573          .   Allocations outstanding
SMA.s0.g_bytes                       1073105012          .   Bytes outstanding
SMA.s0.g_space                           636812          .   Bytes available

I find it hard to believe there is 2.7GB overhead for a 1GB cache, what am I missing here?

Restarting with systemctl restart varnish drops the usage temporarily (as it empties the cache, but also the non-cache memory disappears), but it gradually builds up again to settle at 3.7GB.

This is a test system under somewhat heavy load, is it possible this memory is genuinely in use serving requests? I can see this in the transient space, to my eye this looked uninteresting but after reading some other threads maybe it is.

SMA.Transient.c_req                     9720800         7.90 Allocator requests
SMA.Transient.c_fail                          0         0.00 Allocator failures
SMA.Transient.c_bytes              324751584558    263852.57 Bytes allocated
SMA.Transient.c_freed              324751584558    263852.57 Bytes freed
SMA.Transient.g_alloc                         0          .   Allocations outstanding
SMA.Transient.g_bytes                         0          .   Bytes outstanding
SMA.Transient.g_space                         0          .   Bytes available

I've also seen a few suggestions that varnish is designed for use with jemalloc, I do not have jemalloc installed, it looks like varnish is using glibc's malloc.

EDIT: Full varnishstat -1 output can be found here: https://pastebin.com/3uZ1kpy6, and varnishadm param.show here: https://pastebin.com/7jrLKdMx. At the time of this stat dump, memory usage was at 3GB.

ps -p 917285   -o %mem,vsz,rss
%MEM    VSZ   RSS
12.8 7278920 3136620

Answer 1

malloc cache storage only represents a part of Varnish's memory consumption.

Transient storage

As you mentioned in your question, the transient storage can also consume memory. Transient storage is there to buffer or stream non-cacheable and shortlived content. It is unbounded, which means it could cause an OOM in case of spikes.

FYI: the shortlived runtime parameter defines what shortlived content is and defaults to 10 seconds.

Threads

There is also the runtime cost of Varnish. For every thread there is some memory allocation taking place.

The amount of threads is limited by the thread_pool_min & thread_pool_max runtime parameters per thread pool. By default these values are a minimum of 100 and a maximum of 5000 per thread pool.

Since there are 2 thread pools (as defined by thread_pools), it 'll range between 200 and 1000 if you rely on default values.

The workspace_thread runtime parameter defines the initial amount of memory consumed per thread. The default value is 2 KB.

More workspace memory consumed

Besides the workspace_thread memory consumption per thread, there is additional memory consumption for each thread depending on the type of workload it is processing.

If the thread is used to manage the TCP session, and additional 0.75k of memory is consumed by default, as defined by the workspace_session runtime parameter.

If the thread is used to handle an HTTP request, 96k of memory is consumed by default, as defined by the workspace_client runtime parameter.

And for backend requests the memory consumption is defined by the workspace_backend runtime parameter, which also defaults to 96k.

Tuning

If you altered any of the runtime parameters I mentioned, it's going to have an impact on the total memory consumption.

You are free to tune these parameters accordingly, and its very much a balancing act between making sure you have enough threads to handle all the incoming requests and backend requests.

But also to have enough memory to handle all the logic that you specified in your VCL file.

An alternative solution is the Massive Storage Engine that is part of Varnish Enterprise, the commercial version of Varnish. It has a feature called the Memory Governor, which has the ability to limit the total memory consumption of Varnish, and autoscale the size of the cache depending on the runtime memory it requires.