If you work with HPC clusters, chances are you use slurm every day to submit jobs, monitor queues, and manage compute resources.
Most users know commands like sbatch, squeue, and sinfo, but fewer understand what actually happens internally when a job is submitted.
This article explains how Slurm handles resource allocation behind the scenes, from job submission to execution on compute nodes.
⸻
What Happens When You Submit a Job?
When a user runs:
sbatch job.sh
Enter fullscreen mode
Exit fullscreen mode
Slurm begins a multi step workflow internally.
The main components involved are:
slurmctld → Central controller daemon
slurmd → Compute node daemon
slurmdbd → Accounting database daemon (optional but common)
Scheduler plugin
Select plugin
Cgroups/task plugins
Each component has a specific role in resource allocation.
⸻
Step 1: Job Submission
The sbatch command sends the job request to slurmctld.
The request includes:
Number of nodes
CPUs
Memory
GPUs
Time…