Somaxconn, SYN backlogs, TCP syncookies, and finding our breaking point with stepped Autocannon load benchmarks.

The alert hit our channel at 11:14 AM on a Tuesday.

Our product had just been picked up on a popular tech newsletter, and inbound traffic surged from our normal baseline of 8 requests per second to over 180 requests per second within three minutes.

On the surface, our 2-vCPU, 4GB RAM cloud instance looked relaxed. CPU utilization hovered around 32%. Memory had nearly 2GB free. Disk I/O was barely moving.

Yet visitors were seeing 502 Bad Gateway errors, and API clients reported connection timeouts:

curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to api.example.com:443

Enter fullscreen mode

Exit fullscreen mode

Why would a server drop incoming web traffic when more than half of its compute capacity was sitting idle?

The answer was not in the application code, the database queries, or the cloud provider firewall.…