Menu

1 / 2

Python 3.13 no‑GIL scaling on 64 cores: faster than 3.12 multiprocessing for our data pipeline?

DEV Community·ANKUSH CHOUDHARY JOHAL·7 months ago
#rRtgPwNE
#python#nogil#scaling#cores#multiprocessing
Reading 0:00
15s threshold

Python 3.13 No-GIL Scaling on 64 Cores: Faster Than 3.12 Multiprocessing for Data Pipelines?

Python’s Global Interpreter Lock (GIL) has long been a bottleneck for multi-core scaling, forcing developers to rely on multiprocessing or alternative runtimes for CPU-bound workloads. With Python 3.13’s experimental no-GIL mode (PEP 703), we tested whether removing the GIL delivers better scaling for a real-world data pipeline on 64 cores than Python 3.12’s standard multiprocessing module.

Our Test Setup

We used a bare-metal server with 64 AMD EPYC cores, 256GB RAM, running Ubuntu 22.04 LTS. The test workload was a production-grade ETL pipeline processing 100GB of JSON log data: parsing, filtering invalid entries, aggregating metrics by user ID, and writing results to Parquet.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.