Our reporting service is a Python application that queries a Postgres replica in another network zone, through our data centre firewall. It uses SQLAlchemy's connection pool with the settings it was born with. From about six in the evening until seven in the morning nobody uses it.

In October the depot managers' seven o'clock dashboard sat loading for a quarter of an hour, three mornings running.

The firewall keeps a session for every TCP connection it lets through and removes a session that has been idle for an hour. It removes it silently, sending nothing to either end. Overnight our pool kept five connections open that the firewall had forgotten. In the morning SQLAlchemy handed one to the first request, psycopg2 wrote the query to the socket, and the firewall dropped the packet because it no longer belonged to anything. With no reply and no reset, the Linux kernel did what it should on a lossy network and retransmitted, waiting longer each time, up to the number of attempts in net.ipv4.tcp_retries2.…