Menu

AWS Glue Concurrent Execution: What Happens When Multiple Jobs Write to the Same S3 Location?
📰

AWS Glue Concurrent Execution: What Happens When Multiple Jobs Write to the Same S3 Location?

DEV Community: aws·Rahul R·13 days ago
#UoTw763n
#devtoaws#devto#dev#glue#jobs
Reading 0:00
15s threshold

Imagine you have two AWS Glue jobs running in production.

Both jobs are processing customer data. Both read different source datasets, perform transformations, and write their results to the same S3 location.

Everything works fine when you execute them individually.

But one day, both jobs start running at almost the same time.

One job finishes successfully. The other also reports success.

However, when you query the final data using Athena, some records are missing, some records appear twice, and the output doesn't look the way you expected.

Where did things go wrong?

This is one of the important problems to understand when building production data pipelines with AWS Glue and Amazon S3.

In this article, let's look at what happens when multiple Glue jobs write to the same S3 prefix, why it creates unexpected results, and how to design a safer approach.

1.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.