Our route planning job runs every night and decides which van takes which parcels the next day. In August we gave it its own pool of three nodes, so its memory use could not crowd out anything else, and tainted them with workload=batch:NoSchedule. Only pods with a matching toleration can be scheduled there, which was the point. The job got the toleration. Nothing else did.
In October the job failed at three in the morning and the depots had no routes at six. The pods had already been cleaned up, so we went to our log store. There was nothing from the batch nodes, not that night and not any night since August. Our node dashboards had no data for them either.
Logs and node metrics both come from DaemonSets, one pod per node: Fluent Bit and node exporter. A DaemonSet's pods are scheduled like any others and respect taints.…