In March our order service spent nine minutes failing every write with cannot execute INSERT in a read-only transaction. A pull request merged that morning had pointed it at the reporting replica. Two of us had approved the Terraform plan for that change.

The service runs on ECS, and its task definition is built in Terraform: a list of container definitions passed through jsonencode, environment variables included. One of those variables was the API key for our address lookup provider, read from a Terraform variable marked sensitive. The pull request was a tidy-up that moved shared settings into locals, and during the move the order service's DATABASE_URL was copied from the reporting worker's block a few lines further down.

Terraform carries sensitivity through expressions. Anything derived from a sensitive value is sensitive too, and jsonencode returns one string, so a single secret anywhere inside it marked the entire string.…