feat(resources): heterogeneous job components

[maharajamihir]

Why

Some work needs two machine shapes at once. The concrete case: a CPU node serving environment VMs alongside the GPU node that trains against them (p-doom/reinforcement-learning's OSWorld/CUA-Gym runs do exactly this, via a hand-written het sbatch).

Today a recipe can only describe one shape, so bringing those runs under labctl means splitting them into two runs — which costs the two properties that make the pairing work:

• co-scheduling — the GPUs idle while the CPU side queues. On a busy cluster that's tens of GPU-hours (queue waits here have hit 6h+).
• coupled teardown — a crashed trainer leaves the fleet holding a node with live VMs.

Plus you'd need an out-of-band rendezvous, since a long-lived service doesn't fit [inputs.*] type = "artifact" (artifacts model completed things).

What

[[resources.components]]. Each entry emits a #SBATCH hetjob separator and its own directive block, so one recipe allocates one SLURM heterogeneous job:

[resources]                 # component 0
gpus = 0
cpus = 48
mem  = "256GB"
time = "12:00:00"

[[resources.components]]    # component 1
gpus  = 4
cpus  = 16
mem   = "128GB"
nodes = 1

Job-wide settings (time, qos, account, dependencies, log paths) stay on component 0, per SLURM semantics. The field defaults to empty, so existing recipes render byte-identically — there's a test for that.

SLURM runs the batch script body on component 0 only, so command places work on the others with srun --het-group=N; examples/recipes/hetjob.toml shows the pattern and the docs call it out explicitly.

What didn't need changing

Run state. sacct -j <id> reports one row per component, and scheduler_outcome already aggregates multi-row output for array jobs with exactly the right priority rule (any failure trumps; running outranks succeeded). scancel <id> cancels every component.

Validation

• cargo test — 32 + 9 pass, cargo fmt --check clean.
• New unit tests assert the emitted directives: separator count, per-component --gres/--cpus-per-task/--mem/--nodes, that job-wide --time isn't duplicated, that component 0's own resources survive, that a component's sbatch_extra rejects a #SBATCH prefix like the top-level one does, and that a recipe without components stays het-free.
• End-to-end on our SLURM: a job rendered with this directive syntax allocated two components — sacct reported 140467+0 COMPLETED and 140467+1 COMPLETED.

Note, not included

.gitignore contains only *.log — no target/. A git add -A in a built checkout stages ~5k files including a 123 MB binary (it caught me). Left out to keep this single-concern, but worth a one-liner.

🤖 Generated with Claude Code

[avocadoali]

lgtm