One of the fun things about being a programmer is that as I look more into systems that I didn’t know much about, what superficially appears to be a simple system actually reveals interesting facets of underlying complexity. In other words, reality has a surprising amount of detail . In this post, I want to talk about job queues, which I’ve been thinking about for the past few days (and while drafting this post in my head, I realized I’d thought about them for longer at a previous job, but not nearly with as much clarity.) What do I mean by “job queue”? I mean a system where there is some notion of submitting batch jobs, scheduling them, and running them. Generally, the system is expected to be FIFO or FIFO-like, but that’s not required. Usually, the queue bundles together a native way to schedule jobs periodically – this lets you specify submissions using configuration files (e.g. using JSON or YAML).…