Why Background Jobs?
Fetching trending videos is fast. But validating that 5,000 thumbnails still load, enriching metadata with language detection, and purging broken embeds from the database — these tasks can take minutes. Running them synchronously inside a cron job blocks the fetcher and risks timeouts.
Celery with a Redis broker moves this work off the hot path. TopVideoHub uses a four-worker Celery setup to keep its 9-region video library clean without affecting fetch latency.
Setup
pip install celery redis requests langdetect
Enter fullscreen mode
Exit fullscreen mode
celery_app.py :
from celery import Celery
app = Celery ( ' topvideohub ' , broker = ' redis://localhost:6379/0 ' , backend = ' redis://localhost:6379/1 ' , )
app . conf .…