experiments

A Claude Code skill that scaffolds and manages research experiment projects for academic papers. One command creates a project with a fixed layout (configs, methods registry, dataset registry, JSONL results, plots, paper), and a CLAUDE.md that tells Claude Code how to work inside it: never re-run a result that exists, append rather than overwrite, one module per method, parameters in YAML, plots that only read data, long runs in tmux.

It encodes the lessons from running several paper projects with Claude Code: the rules that stop it from silently overwriting results, refactoring code it was not asked to touch, or launching a queue that deadlocks on a tmux prefix match.

Install

git clone https://github.com/Ambress92/experiments ~/.claude/skills/experiments

No dependencies beyond Claude Code. The scaffolded projects need Python 3 with numpy, pandas, matplotlib, seaborn, pyyaml and tqdm, which are listed in the generated requirements.txt.

Use

/experiments init <name>              create ./<name>/ with the full scaffold, git init, first commit
/experiments add <exp_name>           add an experiment script, its config, a Makefile target
/experiments add-method <method_name> add a method/baseline module and register it
/experiments status                   report methods, datasets, results and figures on disk

Run init from the folder that should contain the project, the others from inside a project created by init.

What init creates

<name>/
  CLAUDE.md               paper metadata, hypotheses, experiment table, and the workflow rules
  README.md
  Makefile                one target per experiment, `make all` runs everything
  requirements.txt
  configs/<exp>.yaml      methods, datasets, seeds, output dir
  configs/method/         optional per-method params, auto-loaded when the experiment YAML says null
  data/                   raw datasets (gitignored, README explains how to obtain them)
  src/utils.py            save/load results, dedup by run ID, seeding, LaTeX tables
  src/data_loading.py     DATASET_REGISTRY
  src/methods/base.py     BaseMethod with setup / run / teardown
  src/methods/__init__.py METHOD_REGISTRY
  src/experiment_*.py     generate | run | evaluate | plot | all
  scripts/run_all.sh      unattended entry point with bounded retry
  scripts/chain_template.sh  one tmux stage waiting on its predecessor
  experiments/<exp>/results.jsonl   universal final output, appended, never overwritten
  plots/                  paper-ready PDFs
  paper/                  LaTeX skeleton

What the rules enforce

• Every result carries a run ID, a hash of (method, params, seed). Runs that already exist are skipped; results files are appended to, never truncated.
• Methods are modules extending BaseMethod and registered in a dict, never if/else branches in experiment scripts. Datasets likewise.
• All tunable parameters live in YAML under configs/.
• Pipeline stages are independent: plot and LaTeX tables only read results.jsonl, they never run anything.
• Run loops catch exceptions per seed and summarise failures at the end.
• Deleting results is selective (delete_results(path, method=, seed=)) and always confirmed with the user.
• Long jobs run in tmux on the machine that owns the compute, one session per stage, matched by exact session name. Each chain script records the predicted effect size and the noise floor in its header before it runs.

Layout

SKILL.md     the skill: subcommands and the full scaffold, read by Claude Code

License

MIT