Using Farm

Farm is the College of Agricultural and Environmental Sciences cluster run by HPC@UCD. It runs Ubuntu 22.04 and the Slurm scheduler, and it's the lab's main computing resource. The official docs are at https://docs.hpc.ucdavis.edu. This page covers what you need to start working. See Farm Tips and Tricks for monitoring jobs and more advanced use, and Farm Software Installation for modules and conda. The lab also uses Hive, a second HPC@UCD cluster with the same /quobyte storage; see Using Hive.

Last checked against the cluster and HPC@UCD docs: September 2026.

1. Get an account

Farm uses SSH keys only; there is no password login.

1. Make an SSH key on your own computer (not on Farm). Use a passphrase.

    ssh-keygen -t rsa -b 4096
   

   This creates ~/.ssh/id_rsa (private: never share or copy it anywhere) and ~/.ssh/id_rsa.pub (public: this is what you upload). HPC@UCD's docs specify RSA keys.

2. Request an account at HiPPO. Log in with your UC Davis credentials, choose Farm, and give Jeff (Ross-Ibarra, jrigrp) as your sponsor. For access type choose SshKey, and also OpenOnDemand if you want the web interface (RStudio, Jupyter, etc.). Paste the contents of id_rsa.pub. HiPPO accepts only one key per upload.

3. Jeff approves the request. Your account is live about an hour after the confirmation email. If you get a new computer, upload a new public key through HiPPO.

Note that the admins and Jeff can access your account if needed; having an account on the cluster constitutes permission for this.

2. Log in

ssh USERNAME@farm.hpc.ucdavis.edu

USERNAME is your UC Davis login ID. (Old addresses like agri.cse.ucdavis.edu and farm.cse.ucdavis.edu no longer work.) If you're asked for a password, your key wasn't accepted. Check that ~/.ssh/id_rsa has permissions 600, or point to the key with ssh -i.

Add this to ~/.ssh/config on your computer so you can just type ssh farm:

Host farm
    HostName farm.hpc.ucdavis.edu
    User USERNAME
    ServerAliveInterval 60

On Windows, use WSL2 or MobaXterm.

What you can do on the login node

The login node (where you land when you ssh in) is shared by everyone and limited to 2 CPUs per user. Use it only for editing files, submitting and checking jobs, small downloads, and light compiling or installing. Anything that uses real CPU, memory, or I/O goes through Slurm, either as a batch job or an interactive session (below). You can't ssh directly to compute nodes.

3. Accounts and partitions: high vs. low

Every job needs an account (-A, who pays) and a partition (-p, which pool of machines).

Always use -A jrigrp. Your default account is publicgrp (the free tier), which can only use low at the lowest priority. If you forget -A jrigrp on high you'll get Invalid account or account/partition combination specified.

The lab has access to two partitions:

	high	low
What it is	Nodes the lab has bought into	All idle resources on the cluster, including big-memory and GPU nodes
Lab limit	608 CPUs and 1,372 GB RAM total, shared across the whole lab	No lab limit
Can your job be killed?	No; it runs until it finishes	Yes. If a high job needs the resources, your job is killed and requeued, restarting from scratch
Max time	150 days	7 days
Nodes	64–256 CPUs, 256–512 GB RAM	Also includes 1–2 TB big-memory nodes and GPUs

When to use which:

• high: long jobs, jobs that can't easily restart, and anything you need done by a deadline. Remember the limits are for the whole lab. Before using more than 100 CPUs or 256 GB RAM on high, check what's in use (Farm Tips and Tricks shows how) and post in #farm_issues on Slack. (No need to post for low.) Memory is usually the binding limit: 1,372 GB ÷ 608 CPUs is only ~2.25 GB per CPU.
• low: large arrays of short jobs, anything that can restart safely, overflow when the lab's high share is full, and anything needing more than ~500 GB RAM or a GPU (--gpus=1 or --gpus=TYPE:1). high has no big-memory or GPU nodes for us. Use -A jrigrp on low too: the lab's low QOS has higher priority than publicgrp.

(Old partitions like med, med2, high2, low2, bigmemh/m/l, and bmm were removed in December 2025. If you see them in an old script, change them.)

To see exactly what you can use:

/opt/hpccf/bin/slurm-show-resources.py --full

4. Batch jobs

Most work runs as a batch script submitted with sbatch. A template:

#!/bin/bash -l
#SBATCH -J myjob                  # job name
#SBATCH -A jrigrp                 # account: always jrigrp
#SBATCH -p high                   # partition: high or low
#SBATCH -t 1-00:00:00             # time limit (D-HH:MM:SS)
#SBATCH -c 4                      # CPUs
#SBATCH --mem=16G                 # memory for the whole job
#SBATCH -o slurm-log/%x-%j.out    # stdout (%x = job name, %j = job ID)
#SBATCH -e slurm-log/%x-%j.err    # stderr
#SBATCH --mail-type=END,FAIL      # optional: email when done/failed
#SBATCH --mail-user=you@ucdavis.edu

set -euo pipefail
module load samtools              # load modules INSIDE the script

samtools sort -@ $SLURM_CPUS_PER_TASK -o out.bam in.bam

Submit it with sbatch myjob.sh. Things to know:

• Always set -t, -c, and --mem. If you don't set memory you get 2 GB per CPU. Jobs that exceed their memory are killed. Tell your program how many threads to use ($SLURM_CPUS_PER_TASK) so it matches -c.
• Time: ask for 1.5–2× what you expect. Jobs are killed at the limit. Shorter requests start sooner, though.
• Log directories must exist before you submit (mkdir -p slurm-log). Slurm won't create them, and the job fails with no output at all if they don't exist.
• Start every script with set -euo pipefail. Without it, "COMPLETED" can lie: only the exit status of the last line counts, so a crashed bwa followed by a successful samtools shows as success.
  • -e: stop at the first failed command.
  • -u: an unset variable is an error, so a typo like $OUTDIIR stops the script instead of silently becoming "" (and rm -rf $OUTDIR/ can't become rm -rf /).
  • -o pipefail: zcat bad.gz | sort | gzip > out.gz fails if any step fails, not just gzip, so you don't get an empty out.gz marked as done.
• Modules are cleared when the job starts, so module load inside the script, not just in your login shell.
• Cancel with scancel JOBID (or scancel -u $USER for all your jobs). Check with squeue --me.

Array jobs

To run the same script on many inputs, use an array instead of submitting hundreds of jobs:

#SBATCH --array=1-100%20          # tasks 1–100, at most 20 running at once

Inside the script, $SLURM_ARRAY_TASK_ID gives the task number, and %A/%a in log names give the array job ID and task ID. A common pattern is to pick the Nth line of a file list:

sample=$(sed -n "${SLURM_ARRAY_TASK_ID}p" samples.txt)
bwa mem -t $SLURM_CPUS_PER_TASK ref.fa ${sample}_R1.fq.gz ${sample}_R2.fq.gz > ${sample}.sam

Use a % throttle on high so one array doesn't eat the whole lab allocation.

5. Interactive work

For exploring data, testing commands, or installing software, get an interactive shell on a compute node:

srun -A jrigrp -p high -t 2:00:00 -c 2 --mem=8G --pty bash -l

Type exit when you're done; the resources are held until you do. Run it inside tmux on the login node so a dropped connection doesn't kill your session (see Farm Tips and Tricks).

RStudio, Jupyter, VS Code, and a desktop run in the browser through Open OnDemand. It needs the OpenOnDemand access type on your HiPPO account. Pick the account (jrigrp), partition, CPUs, memory, and time in the form. These are Slurm jobs too, so close them when you're done.

Farm's OnDemand can't see /quobyte. To work on lab data in RStudio, Jupyter, or VS Code through the browser, use Hive's OnDemand instead (see Using Hive).

6. Storage

Location	What it's for
/home/USERNAME	Small stuff: scripts, configs. The quota is small (20 GB for new accounts). Don't put data or conda environments here.
/quobyte/jrigrp/	Lab storage. All data and projects go here. Make a directory for yourself (/quobyte/jrigrp/USERNAME). Shared lab data is in /quobyte/jrigrp/DATA/ (ACTIVE, ARCHIVED, RESOURCES).
/quobyte/jrigrp/BACKED-UP/	The only backed-up lab space (snapshots via the HPC@UCD backup system). Keep raw data here.
/scratch or /tmp (on compute nodes)	Fast local disk for a running job, deleted automatically when the job ends. See Farm Tips and Tricks.

The old /group/jrigrp* directories and /group/jriscratch are gone. Everything is on /quobyte/jrigrp now.

• Nothing outside BACKED-UP is backed up. Anything deleted there is gone. Keep raw data in BACKED-UP and code in git.

• Full storage means everything dies. Jobs fail cryptically or write truncated files, a full home can block logins and OnDemand, and a full /quobyte breaks things for the whole lab. Check often with df -h /quobyte/jrigrp and ncdu DIR (or du -sh DIR). Compress FASTQs and delete intermediate files. Talk to Jeff before adding more than 1 TB of new data.

• Watch hidden home hogs. ~/.conda and ~/.cache quietly fill your 20 GB home. Point caches at /quobyte in ~/.bashrc:

    export CONDA_PKGS_DIRS=/quobyte/jrigrp/USERNAME/.conda/pkgs
    export APPTAINER_CACHEDIR=/quobyte/jrigrp/USERNAME/.apptainer
    export HF_HOME=/quobyte/jrigrp/USERNAME/.cache/huggingface
  

• Don't have multiple jobs write to the same file on quobyte (e.g. every array task appending to one log). HPC@UCD kills such jobs and may lock the account. Have each task write its own file and combine them afterward.

7. Moving data

From your own computer:

rsync -av --info=progress2 localdir/ farm:/quobyte/jrigrp/USERNAME/project/
scp file.txt farm:/quobyte/jrigrp/USERNAME/

• rsync -a --info=progress2 is the default choice. It's resumable and only sends changes. For big transfers, run it (or wget/curl) on Farm inside tmux so it survives a dropped connection.
• scp for a file or two. The OnDemand file browser upload works for files under ~200 MB. Use Hive's OnDemand for this, since Farm's can't reach /quobyte.
• rclone (module load rclone) moves data between the cluster and Box or Google Drive from the command line.
• Globus is available to everyone (collection "UC Davis Farm home"), but it's still in development.
• Verify important data after a transfer with md5sum or sha256sum, e.g. against the sequencing center's checksums. Verify copies before deleting the source, since only BACKED-UP is backed up.

8. Getting help

1. Check this wiki, the HPC@UCD docs, and ask in #farm_issues on Slack. Someone in the lab has probably hit the same problem.
2. Check the Farm status page for outages and maintenance.
3. Email farm-hpc@ucdavis.edu. This creates a ticket; emails to individual staff aren't handled. Include your username, that it's Farm, your group (jrigrp), the job ID, the exact command and directory, and the full error text. Attachments are often stripped, so give file paths instead. Support hours are 8–5 on work days. Be patient, they support many clusters.