Menu

writing a free guide on running spark in production on GCP. what am I missing?

Reading 0:00
15s threshold

been doing data engineering for a long time now, mostly Spark for the last stretch. On-prem Hadoop first, then Dataproc, some Databricks, lately Spark on GKE. Everything from boring nightly ETL to multi-terabyte jobs where the shuffle is most of the runtime and the SLA is tight.

I keep explaining the same stuff to new people on the team, and at some point figured I should write it down. So I'm turning it into a free guide.

Not a DataFrames intro, there are plenty of those. More the stuff you only learn after a job that ran fine for months starts crawling and nobody touched it. Most of what I know came from staring at the Spark UI at bad hours, and very little of it is in the Dataproc docs.

Rough list so far:

reading the Spark UI and event logs without guessing, including setting up a persistent history server so the logs outlive the cluster

shuffle partitions and AQE, and the cases where the defaults make things worse

skew.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.