-- The "Anti-Pattern": Thousands of tiny files in a deep partition structure SELECT count (*) FROM raw_logs WHERE event_date = '2023-10-01';
-- The "Optimized" Approach: Compaction and partition pruning SELECT count(*) FROM optimized_logs WHERE event_date = '2023-10-01' ;
Enter fullscreen mode
Exit fullscreen mode
Why this matters: Amazon Athena charges based on the volume of data scanned. When your S3 layout creates a "small-files problem," Athena is forced to open, read metadata, and scan thousands of tiny objects, leading to increased latency and unnecessary costs that scale linearly with your technical debt.
Every data engineer working with AWS eventually reaches the same threshold: the moment their S3 bucket transitions from a simple landing zone to a queryable data lake. You start with a few JSON files, but within months, you have millions of objects spread across a deep, nested directory tree.…