Your upstream source sends a cumulative, month-to-date file : every

delivery carries the whole month so far. Sometimes it arrives once a

month, sometimes every day. Your pipeline lands it in Amazon S3 as

Parquet, partitioned by day, and then builds stage and analytics layers

with Amazon Athena.

Two problems show up quickly:

Reruns duplicate data. Run the same day twice and Athena returns the rows twice.

Every delivery reprocesses the whole month. Thirty days, one Athena query per day per layer, even though most days did not change at all.

This post shows a small pattern that fixes both: an idempotent write per

partition, plus a content signature per partition that lets each

layer skip the days whose input did not change.

1.…