been doing data engineering for a long time now, mostly Spark for the last stretch. On-prem Hadoop first, then Dataproc, some Databricks, lately Spark on GKE. Everything from boring nightly ETL to multi-terabyte jobs where the shuffle is most of the runtime and the SLA is tight.
I keep explaining the same stuff to new people on the team, and at some point figured I should write it down. So I'm turning it into a free guide.
Not a DataFrames intro, there are plenty of those. More the stuff you only learn after a job that ran fine for months starts crawling and nobody touched it. Most of what I know came from staring at the Spark UI at bad hours, and very little of it is in the Dataproc docs.
Rough list so far:
reading the Spark UI and event logs without guessing, including setting up a persistent history server so the logs outlive the cluster
shuffle partitions and AQE, and the cases where the defaults make things worse
skew.…