Menu

1 / 2

Python Celery Task Queues for Video Metadata Processing

DEV Community·ahmet gedik·6 months ago
#ppdUCJQY
#task#celery#python#taskqueue#video_id
Reading 0:00
15s threshold

Why Background Jobs?

Fetching trending videos is fast. But validating that 5,000 thumbnails still load, enriching metadata with language detection, and purging broken embeds from the database — these tasks can take minutes. Running them synchronously inside a cron job blocks the fetcher and risks timeouts.

Celery with a Redis broker moves this work off the hot path. TopVideoHub uses a four-worker Celery setup to keep its 9-region video library clean without affecting fetch latency.

Setup

pip install celery redis requests langdetect

Enter fullscreen mode

Exit fullscreen mode

celery_app.py :

from celery import Celery

app = Celery ( ' topvideohub ' , broker = ' redis://localhost:6379/0 ' , backend = ' redis://localhost:6379/1 ' , )

app . conf .…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.