Menu

Marketplace Reconciliation: Long Background Jobs, Queue Workers, and Cron Timeouts
📰

Marketplace Reconciliation: Long Background Jobs, Queue Workers, and Cron Timeouts

DEV Community: python·MordecaiNilsson7582·5 days ago
#7pX2y4bo
#devtopython#devto#dev#chunk#provider
Reading 0:00
15s threshold

Short answer: make the cron trigger enqueue a small, idempotent reconciliation task, then let workers process checkpointed chunks instead of trying to finish the payment-provider scan inside one 15-minute execution.

That decision keeps the failure boundary visible. A scheduler starts a run. A queue delivers bounded work. A database records what was applied. The worker can stop after one chunk and continue from a durable cursor, which is much easier to test than a long loop that only knows it ran for 14 minutes.

This is the shape I use for notebook-to-prod transitions and eval-driven systems: the run identity, input revision, and progress marker are data, not assumptions hidden in process memory. Prompt-cost awareness matters here too.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.