In September we moved invoice reminders from an old CronJob in our billing namespace to the new billing service, which sends them from its own scheduler. The release was two pull requests: one turned the new reminders on, and one deleted the CronJob's manifest from the billing repository. Both merged on the same afternoon. Argo CD synced, the new reminders started, and we closed the ticket.
Six weeks later a customer forwarded us two reminders for the same invoice, sent on the same morning eleven minutes apart and worded slightly differently. One came from the new service. The other came from a CronJob we had deleted.
The billing Application had automated sync turned on, which we had read as Argo CD making the cluster match Git. It does, for everything that is in Git. Automated sync does not delete a resource that has disappeared from Git unless prune is also turned on, and prune is off by default, as a safety measure, so that a mistake in a repository cannot quietly delete production resources.…