Almost every Azure Databricks project I've seen starts the same way: files land in ADLS Gen2, and something has to pick up only the new ones, reliably, without reprocessing yesterday's data. Auto Loader plus Structured Streaming is the standard answer, and it's also a favourite interview topic because the details (checkpoints, schema evolution, exactly-once) separate people who've run it in production from people who've only read the docs.

Here's a practical walkthrough with PySpark code you can drop into a notebook.

1. The minimal Auto Loader pipeline

from pyspark.sql import functions as F

source_path = " abfss://[email protected]/orders/ " checkpoint = " abfss://[email protected]/bronze_orders/ "

raw = ( spark . readStream . format ( " cloudFiles " ) . option ( " cloudFiles.format " , " json " ) . option ( " cloudFiles.schemaLocation " , checkpoint + " _schema " ) . option ( " cloudFiles.inferColumnTypes " , " true " ) .…