As a data analyst, my job is to get data from A to B, cleaned and ready for use. A common workflow for my team involves users uploading Excel files to a OneDrive folder. A Power Automate flow then syncs these files daily to a container in our Azure Storage Account .

From there, my responsibility begins:

Read the new Excel file from Blob Storage using Python.

Process the data (clean, transform, apply business logic).

Write the final data to an Azure SQL Database.

I needed this to run on two triggers: a time schedule (e.g., every morning at 7 AM) and an event-driven trigger (i.e., as soon as a new file lands in the container).

My first thought was to use the "big data" tools I'd heard of: Azure Databricks or Azure Synapse Analytics .

The "Big Tool" Trap

On the surface, Databricks and Synapse are perfect.

They let me write Python in a Notebook , which I'm very comfortable with.

They have easy-to-use trigger and monitoring tools.

I set up a proof-of-concept, and it worked.…