how to reach 1M concurrent TCP connections?

Question

I am trying to setup a environment with multiple nodes (1 receiver + 16 senders) to establish 1,000,000 (1M) concurrent TCP connections.

Using this tool: https://github.com/Microsoft/ntttcp-for-linux

Using Ubuntu 1804 with latest kernel

Currently, I am easily reached 262,156 concurrent TCP connections with 3+ senders. but after this, seems that the receiver is not able to accept any new TCP connections.

Is there any configuration on Linux I can tune to make 1M TCP connections?

What I have done:

• Receiver side:

ulimit -n 1024000
./ntttcp -P 64 -M -e    ### '-e' to use epoll()

I run this command on receiver side to monitor the number of TCP connections:

ss -ta | grep ESTA | wc -l

• Sender sides

ulimit -n 1024000
echo 1024 65535 > /proc/sys/net/ipv4/ip_local_port_range
./ntttcp -s10.0.0.4 -P 64 -n 10 -l 100    ### 64,000 concurrent TCP connections from one sender

Answer 1

Here are some suggestions, please double check with your available resources. Incrase your socket descriptors

ulimit -n 20000500

increase the maximum number of socket descriptors per process to a value larger than the current kernel maximum (fs.nr_open)

echo 20000500 > /proc/sys/fs/nr_open

ephemeral ports from 500 to the maximum theoretical limit 65,536 – keeping only the ports from 1 to 500 reserved for the operating system

sysctl -w net.ipv4.ip_local_port_range="500   65535"

Client socket descriptor change

echo 3000000 > /proc/sys/fs/nr_open
ulimit -n 2000000

echo 1 > /proc/sys/net/ipv4/tcp_tw_recycle
sysctl -w net.ipv4.tcp_mem="383865   511820   2303190"
sysctl -w net.ipv4.tcp_rmem="1024   4096   16384"
sysctl -w net.ipv4.tcp_wmem="1024   4096   16384"
sysctl -w net.ipv4.tcp_moderate_rcvbuf="0"