compitum

Status quick links:

• CI: https://github.com/PaulTiffany/compitum/actions/workflows/ci.yml
• Docs: https://github.com/PaulTiffany/compitum/actions/workflows/docs.yml
• Notebooks: https://github.com/PaulTiffany/compitum/actions/workflows/notebooks.yml
• Full Validation (manual/scheduled): https://github.com/PaulTiffany/compitum/actions/workflows/full.yml
• Mutation (PR label): https://github.com/PaulTiffany/compitum/actions/workflows/mutation_on_label.yml
• Mutation Dispatcher (manual/scheduled): https://github.com/PaulTiffany/compitum/actions/workflows/mutation_dispatch.yml

A production-ready, geometrically-aware AI router with SPD metric learning, constraint-aware selection (shadow prices), metric-aware KDE coherence, and Lyapunov-stable online updates.

Install

python -m venv .venv && source .venv/bin/activate
pip install -e .

Quick demo

compitum route --prompt "Prove the binomial identity using generating functions."

Run tests

Quick run:

pytest

CI‑parity run (mirrors CI deselections and markers):

make test-ci

For Hypothesis settings parity with CI:

• PowerShell: $env:HYPOTHESIS_PROFILE='ci'; pytest -q
• Bash: HYPOTHESIS_PROFILE=ci pytest -q

See configs/ and examples/ for constraints and a synthetic benchmark.

Community

• Philosophy: PHILOSOPHY.md
• Contributing: CONTRIBUTING.md
• Code of Conduct: CODE_OF_CONDUCT.md
• Security: SECURITY.md
• Support: SUPPORT.md
• Workflows: WORKFLOWS.md

Release Artifacts

• Consolidated report: reports/report_release.html
• Certificate schema (JSON): docs/_extra/assets/certificate.schema.json

Examples

• Docs page: docs/Examples.md
• Folder on GitHub: https://github.com/PaulTiffany/compitum/tree/main/examples
• In-repo overview: examples/README.md

Notebooks

• Folder: notebooks/
• Starter: notebooks/Getting_Started.ipynb
• Setup: notebooks/README.md
• Wiki walkthroughs: https://github.com/PaulTiffany/compitum/wiki

Matbench (Offline)

• Docs: docs/Matbench.md
• CLI: tools/eval_matbench_srmf.py (offline CSV adapter). Conservative, reproducible outputs.

Matbench Regret (Offline)

• Docs: docs/Matbench-Regret.md
• Calibration CLI: tools/calibrate_matbench_srmf.py
• Regret CLI: tools/eval_matbench_regret.py

Core Science 0.2.0

• Geometry: SPD bounds, triangle inequality, ray monotonicity, update descent.
• Stability: Lyapunov decay/saturation/recovery; ΔV proxy sequences; combined update boundedness.
• Coherence: monotone outward, ±v symmetry, inward score direction, mixture discrimination.
• Constraints: feasibility monotone; duals slack ≈ 0, boundary ≈ 0; monotone/scale sanity.
• Determinism: repeated/batch determinism; paraphrase flip budget + explainability.
• Pedagogy: practice raises evidence/utility (beta_s > 0); prepared environment fixes constraints.

Run invariants

pytest -q tests/invariants           # smoke
pytest -q -m lg                      # geometry/learning
pytest -q -m cl                      # explainability/determinism
pytest -q -m sy                      # control/stability
pytest -q -m stat                    # estimation/uncertainty
pytest -q -m pedagogy                # control of error

RouterBench Data (5-shot pickle)

Some RouterBench‑based scripts expect a local copy of routerbench_5shot.pkl (not redistributed here).

• Download from: https://huggingface.co/datasets/withmartian/routerbench/blob/main/routerbench_5shot.pkl
• Or use the resolve URL in the fetch script (recommended):

python scripts\fetch_routerbench.py --also-copy-to-src

This places the file at data/routerbench_5shot.pkl and also copies it to src/routerbench/routerbench_5shot.pkl for compatibility with existing defaults. You can provide --sha256 <HEX> to verify integrity.

Security note: .pkl files can execute code when loaded; download only from trusted sources.

Testing Strategy

The project maintains a rigorous, deterministic testing program.

• CI Profile (default): pytest runs with HYPOTHESIS_PROFILE=ci. This uses a fixed random seed and a moderate number of examples (max_examples=100) for fast, repeatable builds.
• Mutation Profile: For mutation testing with mutmut/cosmic-ray, a dedicated HYPOTHESIS_PROFILE=mutation_ci is used via scripts/cr_pytest.sh. This allows for a different number of examples to balance thoroughness and speed.
• Invariants Suite: A dedicated property-based test suite in tests/invariants/ validates the core mathematical and operational invariants of the system. These tests are marked with @pytest.mark.invariants.

Locally, make check runs lint (ruff check .) + mypy -p compitum --ignore-missing-imports --hide-error-context + bandit + pytest; CI additionally runs mypy --strict --disable-error-code no-any-return src/compitum and ruff format --check as separate jobs. All of these currently pass clean. Mutation testing itself is advisory and dispatch/label-gated, not part of local make check — see WORKFLOWS.md for how to trigger the Mutation Dispatcher/Mutation PR Label workflows, or run bash scripts/cr_pytest.sh directly against a local cosmic-ray session.

Export Control

This project is open-source research code (MIT). Use is subject to U.S. export laws and sanctions compliance. Do not use if you are a sanctioned person/region.

Benchmark Profiles

Three run profiles are provided for benchmarking. They are configured via environment variables.

• SMOKE: A very fast run to ensure the benchmarks execute without error.

  set COMPITUM_SKIP_PERF_ASSERTIONS=1 && set COMPITUM_STEPS=400 && set COMPITUM_REFIT_POLICY=never && set COMPITUM_UPDATE_BATCH_SIZE=100000 && pytest -q -k "test_energy_drift or test_constraint_violation_rate or test_spd_det_and_trust_radius_bounds" --benchmark-min-time=0.01

• DEV: A standard development run for quick performance checks.

  set COMPITUM_STEPS=800 && set COMPITUM_REFIT_POLICY=adaptive && set COMPITUM_UPDATE_BATCH_SIZE=4000 && pytest -q -k "test_energy_drift or test_constraint_violation_rate or test_spd_det_and_trust_radius_bounds or test_iso_utility_savings_vs_fixed_best or test_router_throughput_and_latency" --benchmark-autosave --benchmark-save=DEV

• FULL: The complete diagnostic run, including the heavy benchmarks.

  set COMPITUM_STEPS=4000 && set COMPITUM_REFIT_POLICY=periodic && set COMPITUM_UPDATE_BATCH_SIZE=2000 && pytest -q -m heavy_bench --benchmark-autosave --benchmark-save=FULL

A Note on Threading

For performance-critical benchmark runs, it is advisable to set the following environment variables to prevent thread over-subscription from libraries like NumPy, which can interfere with measurements:

set OMP_NUM_THREADS=1
set MKL_NUM_THREADS=1
set OPENBLAS_NUM_THREADS=1
set NUMEXPR_NUM_THREADS=1

Running RouterBench Evaluation

To run the full routerbench evaluation, follow these steps:

1. Setup RouterBench Environment

Create a new virtual environment for routerbench and install the dependencies from src/routerbench/requirements.txt.

python -m venv .venv-routerbench
.venv-routerbench\Scripts\activate
pip install -r src/routerbench/requirements.txt

2. Pre-train Predictors

For fast evaluation, it's crucial to pre-train the CalibratedPredictor models to avoid lengthy fitting times during each run.

Run the pretrain_predictors.py script using the routerbench virtual environment:

set PYTHONPATH=%CD%\src && .\.venv-routerbench\Scripts\python.exe -m scripts.pretrain_predictors

This will save the pre-trained predictors to data/pretrain_predictors/predictors_all-MiniLM-L12-v2_0.1.joblib.

3. Run the RouterBench Evaluation

Execute the evaluate_routers.py script as a module within the routerbench package. This command will run the full evaluation and generate the results.

set PYTHONPATH=%CD%\src && .\.venv-routerbench\Scripts\python.exe -m routerbench.evaluate_routers --config data\routerbench\evaluate_routers.yaml --local --data-path routerbench_5shot.pkl

This command will generate CSV and PKL files in the data/eval_results directory, containing the evaluation metrics for various router models.

Dev Tips

• Hypothesis CI parity locally:
  • PowerShell: $env:HYPOTHESIS_PROFILE='ci'; pytest -q
  • Bash: HYPOTHESIS_PROFILE=ci pytest -q

Educator Pack

Generate a small, self-contained pack for workshops:

python scripts/generate_classroom_pack.py

This writes artifacts/pedagogy_pack.zip with the lab worksheet (docs/Pedagogy-Lab.md), a demo script, a sample certificate JSONL (if available), and a tiny prompt set.

Binder (Supercon Quickstart): https://mybinder.org/v2/gh/PaulTiffany/compitum/HEAD?labpath=notebooks%2Fexamples%2FSupercon_Quickstart.ipynb