Imagine building an AWS Glue job that processes sales files from Amazon S3 every morning.

On Monday, the job processes 10 files. On Tuesday, five new files arrive.

You want the next job run to process only those five new files rather than reading all 15 again.

But how does AWS Glue know which files it has already processed?

That's where AWS Glue Job Bookmarks come in.

In this guide, we'll explore how AWS Glue Job Bookmarks work, how to enable them, and how to build a simple incremental ETL pipeline using PySpark.

What Are AWS Glue Job Bookmarks?

AWS Glue Job Bookmarks help supported ETL jobs remember which source data has already been processed.

When a job runs successfully and saves its bookmark state, subsequent runs can identify new or modified source data rather than repeatedly processing everything.

For supported Amazon S3 sources, AWS Glue uses object modification timestamps to determine which files need processing.

For example:

Monday: Your job processes 10 CSV files.

Tuesday: Five new CSV…