Menu

Scheduled Marketplace Cleanup — Quota-Paced Workers Delete Stale Uploads After Webhooks
📰

Scheduled Marketplace Cleanup — Quota-Paced Workers Delete Stale Uploads After Webhooks

DEV Community: node·DrummondReed8257·5 days ago
#qtjrrrRL
#devtonodejs#devto#dev#queue#cleanup
Reading 0:00
15s threshold

Short answer: run one nightly schedule that finds stale upload IDs and enqueues them, then let a small worker pool delete at a controlled pace with an idempotent delete path. Do not make the scheduled request perform the whole cleanup.

For a marketplace, I would choose by recovery behavior before convenience:

Choice Schedule boundary Rate control Recovery trade-off

Infrai cron plus queue Public HTTP trigger enqueues IDs Worker code or consumer concurrency At-least-once delivery requires repeat-safe deletes

PostgreSQL with FOR UPDATE SKIP LOCKED An external scheduler starts a database worker Worker count and application pacing Few moving parts, but queue load shares the primary database

Celery Scheduler hands work to Celery workers Worker-side controls A specialist worker system brings its own operating surface

Temporal or Airflow Workflow orchestration owns the run Workflow and activity controls Better for multi-step orchestration; more machinery for a delete sweep

My default: for an EU SaaS already…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.