---
title: "Slurm jobs"
description: "Write a Slurm job that asks for a GPU and runs in a container on a DGX Spark."
---

> Documentation Index
> Fetch the complete documentation index at: https://docs.c4e.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Slurm jobs

Every job on the cluster runs through [Slurm](https://slurm.schedmd.com/documentation.html). Each Spark has one GB10 GPU, so a job asks for at most one GPU per node.

## A first GPU job

```bash
#!/bin/bash
#SBATCH --job-name=gpu-check
#SBATCH --partition=jobs
#SBATCH --gres=gpu:1
#SBATCH --cpus-per-task=8
#SBATCH --mem=64G
#SBATCH --time=01:00:00

srun --container-image=nvcr.io#nvidia/cuda:13.0.1-base-ubuntu24.04 nvidia-smi -L
```

Submit it from [Open OnDemand](https://docs.c4e.ai/getting-started/ondemand/)'s Job Composer. The output file should list one GB10 GPU.

## Ask for memory

> **Always set --mem**
>
> On the GB10 the CPU and GPU share one pool of memory, and GPU memory counts against your job's limit. Without `--mem`, Slurm gives a job 1 GB per CPU, and GPU work runs out of memory almost at once.

Each Spark has 128 GB in total, and the system keeps some of it. Asking for up to about 100 GB on one node is safe.

## Containers

Jobs run containers through NVIDIA's [Pyxis](https://github.com/NVIDIA/pyxis) and [Enroot](https://github.com/NVIDIA/enroot). Add `--container-image` to `srun`:

- `--container-image=ubuntu:24.04` pulls from Docker Hub.
- `--container-image=nvcr.io#nvidia/pytorch:<tag>` pulls from NVIDIA's NGC catalog. Note the `#` after the registry name.

The Sparks use Arm (aarch64) CPUs, so pick images built for Arm. NVIDIA's NGC images for CUDA and PyTorch include Arm builds.

## Limits

| | |
|---|---|
| Partition | `jobs` (the default) |
| Longest run | 7 days (`--time=7-00:00:00`) |
| GPUs | 1 per node (`--gres=gpu:1`) |
| CPUs | up to 18 or 20 per node |

## Jobs pause the shared model

The model behind [chat](https://docs.c4e.ai/getting-started/chat/) and [the API](https://docs.c4e.ai/getting-started/api/) runs as a Slurm job of its own, in an admin-only `serve` partition, on both Sparks. Your jobs come first: when a job in `jobs` needs those nodes, Slurm stops the model and puts it back in the queue. While the cluster has two Sparks, that's almost any job.

> **Your job takes chat and the API offline for everyone**
>
> From the moment your job starts until the model has restarted, chat.c4e.ai and api.c4e.ai can't answer. The model restarts by itself when your job ends, and needs a couple of minutes to load. A 14-second test job kept it offline for about two minutes.

While the model is paused, chat messages and API calls fail with an error instead of an answer. Nothing is lost: wait a few minutes and try again. If you call the API from code, retry with a delay rather than in a tight loop.

To keep the pause short:

- Ask for the time you need with `--time`, not the 7-day maximum. Slurm ends the job at that limit, so the model comes back on time.
- Test with small, short runs before a long one.
- Cancel jobs you no longer want with `scancel <jobid>`.
- Run long jobs when the cluster is quiet. `#SBATCH --begin=22:00` holds a job until 10 PM, and `--begin=now+2hours` works too.

### Check what's running

[Open OnDemand](https://docs.c4e.ai/getting-started/ondemand/)'s **Active Jobs** shows the whole queue. From a shell, run:

```bash
squeue
```

A `tensorfold-serve` job in the `serve` partition marked `R` means the model is running. `PD` means it's waiting for jobs to finish. Your own jobs, and how long they ran:

```bash
squeue --me
sacct -X --format=JobID,JobName,Partition,State,Start,End,Elapsed
```

Once the four new Sparks arrive, the model will move to nodes of its own and jobs will stop pausing it.

Source: https://docs.c4e.ai/getting-started/slurm-jobs/index.mdx
