Menu

CloudWatch had a slow hour and our delivery slot service stopped because it could not finish writing a log line
📰

CloudWatch had a slow hour and our delivery slot service stopped because it could not finish writing a log line

DEV Community: docker·Sergey Shinder·17 days ago
#1QfEDbbA
#devtodocker#devto#dev#service#every
Reading 0:00
15s threshold

In February our delivery slot service stopped answering for fifty minutes. Checkout calls it to offer customers a delivery time, and without it nobody could complete an order. CPU was idle, memory was flat, and every process was alive. Every request thread was waiting on the same thing, a write to standard output.

The service is one of the last we run on plain Docker, on four EC2 hosts, with the awslogs logging driver sending container output straight to CloudWatch Logs. That afternoon CloudWatch Logs in our region was degraded, and calls to put log events were slow and often failing.

Docker's logging drivers have a delivery mode, and the default is blocking. The daemon reads what the container writes and hands it to the driver directly, and when the driver cannot keep up, the daemon stops reading. The pipe between the container and the daemon holds sixty four kilobytes, and once it is full a write to stdout simply does not return. Our service logs a line for every request.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.