Menu

1 / 2

Python Celery Task Queues for Video Metadata Processing

DEV Community·ahmet gedik·6 months ago
#kZlV55hM
#task#python#celery#taskqueue#video_id
Reading 0:00
15s threshold

When Synchronous Processing Breaks Down

As DailyWatch grew past 50,000 videos across 8 regions, our cron pipeline started hitting the wall. Thumbnail validation, metadata enrichment, and broken link detection can't all run in a single sequential script without blowing past timeout limits. The solution: offload heavy work to Celery task queues.

Setting Up Celery with Redis

Install the dependencies:

pip install celery[redis] requests pillow

Enter fullscreen mode

Exit fullscreen mode

Configure Celery with sensible defaults:

# celery_app.py from celery import Celery

app = Celery ( " dailywatch " , broker = " redis://localhost:6379/0 " , backend = " redis://localhost:6379/1 " , )

app . conf .…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.