Menu

1 / 5

Use dedicated IAM roles for every data pipeline job

DEV Community: aws·Aniket Abhishek Soni·about 1 hour ago
#1hIJ195v
#devtoaws#devto#dev#ccf#eee
Reading 0:00
15s threshold

Most data engineers assume that a single "ETL-Role" shared across an entire orchestration platform is sufficient because the services are internal. This assumption is a common precursor to lateral movement if a single job is compromised, as that job inherits the broad permissions of every other pipeline in your environment. When a job that only needs to read from a specific S3 prefix suddenly has the ability to delete tables in your data warehouse or read sensitive secrets from Parameter Store, your blast radius is effectively the entire platform.

This article shows how to break the "shared service role" pattern by implementing job-specific IAM roles. We will focus on defining discrete execution roles for individual tasks, ensuring that a failure in one pipeline component cannot be used to escalate privileges or access data outside of that component’s narrow scope.…

Continue reading — create a free account

Join HashtagPLUS to read full articles, follow hashtags, vote, and join the conversation.