Skip to content

feat(challenges): load docker challenges and move bounty out - #312

Merged
Mathis (echobt) merged 4 commits into
mainfrom
feat/challenge-containers
Sep 24, 2026
Merged

Mathis (echobt) merged 4 commits into
mainfrom
feat/challenge-containers

Conversation

@echobt

@echobt Mathis (echobt) commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Problem and behavior

Before this change, every challenge lived inside the master process, and adding one meant changing Cortex code. Bounty was one of those in-process challenges.

New behavior

  • Challenge containers. The validator master loads any challenge that implements docs/CHALLENGES.md as a Docker container, and keeps it up to date:
    • An unsigned operator registry, deploy/challenges/registry.toml, decides what runs.
    • The owner-signed trust root still decides emission.
    • Once per epoch, the master calls the container's authenticated GET /internal/v1/get_weights?epoch=, signs exact leaves for the sealed participant set, and seals.
    • Public routes are proxied under /challenge/<id>/. internal/ paths, dot segments, % and \ are refused; only 3 headers are forwarded; bodies are capped.
    • GET /v1/metagraph/latest gives containers the sealed hotkey→uid map.
  • Auto-updater: cortex challenge-supervisor. It is the only service with the Docker socket, and it reads no secret. It:
    • pulls stable, edge or a pinned digest from GHCR;
    • checks the slug, contract and source labels;
    • checks the GitHub build provenance;
    • runs a secretless canary on a tmpfs /data;
    • replaces the container, rolls back on failure and remembers the refused digest;
    • removes unregistered containers but keeps their volumes.
  • Algorithm 3 (trust document version >= 3):
    • accepts 1..64 unique challenge ids summing to 10000 bps;
    • each challenge pays share * min(sum(leaves), 10^12) / 10^12, and the shortfall burns to UID0;
    • under algorithm 3, Bounty with full_share_mass = 10 pays exactly what algorithm 2 pays; the E2E test checks this.
    • Algorithms 1 and 2 and every frozen vector are unchanged. Validators still only recompute signed leaves; they need no registry, Docker or token.
  • Bounty moved out. It now lives in CortexLM/bounty:
    • v1.0.0 is published as ghcr.io/cortexlm/bounty:stable and attested;
    • its SQLite schema and session HMAC are unchanged, and its operator guide migrates bounty.sqlite3 and the session key.
    • Removed from Cortex: src/cortex/bounty, docs/BOUNTY.md and the BOUNTY_* master settings.
  • OpenType. OpentypeAI/challenge is the second container, registered in the example registry. It stays unpaid until an owner signs a v3 trust root that includes it (burn-in).

Validation

  • Ruff format and lint
  • Mypy
  • Offline test suite: 1074 passed.
    • New unit tests run the supervisor against a fake Docker Engine: rollout, canary without secrets, channel update, rollback, refused digest, bad labels or provenance, pin, restart, prune.
    • New tests also cover registry validation, exact leaf_scores and proxy refusals.
    • E2E runs a Bounty container under algorithms 1, 2 and 3, plus a v3 setup with three containers where the failing and unregistered ones burn.
  • scripts/check_repo.py --final
  • Deployment contracts. check_deploy now allows exactly one Docker-socket service (the supervisor, which mounts nothing else) and requires the gateway to mount the registry and join cortex-challenges.
  • Wheel build
  • Manual: the real supervisor ran against the local Docker daemon with the Bounty image.
    • The canary passed and the container rolled out with a read-only root, CapDrop ALL, no-new-privileges, UID 65532 and no host port.
    • /version returned contract 1; get_weights returned 401 for a wrong bearer and 503 fail-closed with no feed configured.
    • Pruning removed the container and kept the volume.

Public contract

  • Miner documentation is updated: external-miner/bounty.md points to CortexLM/bounty, and the validator guide covers algorithm 3.
  • No challenge content, credential, floating production image, or invented digest was added. The v3 template is unsigned and uses a placeholder epoch.
  • Frozen protocol specifications and BASE_*/domain compatibility remain intact.

Greptile

  • Greptile reviewed this PR and findings are resolved or answered
  • If the bot was silent, I commented @greptileai review

Risk

  • Emission. Nothing changes until an owner signs a version >= 3 document. The committed trust root is still the legacy one.
  • Bounty cutover. Before starting the new master, the operator must follow the migration in CortexLM/bounty docs/operator.md. Until then the Bounty share burns; it is never reassigned.
  • Docker socket. The socket is host-root equivalent. It is confined to the secretless supervisor, and the deploy check enforces that.
  • Known limit (ponytail:). Provenance is an existence check through the GitHub attestations API, not a Sigstore signature verification. The upgrade path is documented.
  • Rollback. Redeploy the previous Cortex image digest together with the old BOUNTY_* env. The Bounty database is copied, not moved.

🤖 Generated with Claude Code

RetriggerConfidence Score: 5/5

No outstanding finding blocks merging.

Summary

The PR moves Bounty into a challenge container and adds container supervision, proxying, and owner-activated algorithm 3. Temporary GitHub attestation failures now retry on a later poll instead of permanently refusing an update. No new actionable issue was identified.

Reviews (4) · Last reviewed commit: "fix(challenges): retry attestation outag..."

The master now runs any challenge that implements docs/CHALLENGES.md as
a Docker container: it reads authenticated get_weights once per epoch,
signs exact leaves, seals, and proxies public routes under
/challenge/<id>/. The new challenge-supervisor is the only process with
the Docker socket. It pulls stable/edge/pinned GHCR images, checks labels
and GitHub build provenance, canaries without secrets, and rolls back.

Trust root version >= 3 enables algorithm 3: 1..64 challenges, each
paying share * min(sum(leaves), 10^12) / 10^12, with the rest burned.
Validators keep verifying signed leaves only. Bounty under algorithm 3
with full_share_mass = 10 pays exactly the algorithm 2 amount.

The in-process bounty service is removed; it now lives in CortexLM/bounty.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@echobt

Copy link
Copy Markdown
Contributor Author

Greptile (@greptileai) review

Comment thread src/cortex/challenges/supervisor.py Outdated
Comment thread src/cortex/challenges/supervisor.py Outdated
Comment thread src/cortex/challenges/client.py Outdated
Comment thread src/cortex/challenges/client.py Outdated
@greptile-apps

This comment has been minimized.

- Rollout now stops the running container and keeps it aside. It removes
  that container only after the replacement answers /version. A failed
  create, start or readiness check restarts the previous container unchanged.
- Each container carries a fingerprint of its full spec. A registry change
  (env, resources, limits) now redeploys even when the digest stays the same.
- An algorithm 2 answer with any weight that is not a u64 integer is
  rejected as a whole, so the challenge burns instead of partly paying.
- Algorithm 1 signed Bounty's legacy champion lattice, which containers
  never compute, so container challenges burn under it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@echobt

Copy link
Copy Markdown
Contributor Author

All four findings, including the outside-diff list, are addressed in fe08ade. I answered each one inline:

  • Rollout failure restores the same serving container. Tested with the fake engine and on real Docker.
  • Registry changes with the same digest now redeploy, through a spec fingerprint label.
  • Algorithm 1 never signs a container weight. The share burns; production is on algorithm 2.
  • One non-u64 weight now rejects the whole answer, so the challenge burns instead of partially paying.

Verification: 1079 tests pass, and ruff, mypy, check_repo --final and check_deploy are green.

Greptile (@greptileai) review

Comment thread src/cortex/challenges/supervisor.py
Comment thread src/cortex/challenges/supervisor.py
- A supervisor restart after it set the serving container aside no
  longer deletes that backup. The next reconcile first restores it.
- Labels and build provenance are re-checked against the current registry
  entry for every new container spec. A source edit with the same digest
  can no longer deploy an image that was never verified against that
  source. Only the canary is skipped for configuration-only changes.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@echobt

Copy link
Copy Markdown
Contributor Author

Both new findings are fixed in 0f80ec9, and I replied and resolved each thread inline:

  • Interrupted rollout: if the supervisor restarts mid-rollout, the next reconcile restores the set-aside container first.
  • Same-digest config changes: these now re-check labels and provenance against the current entry, so a source edit is verified. Only the canary is skipped.

1081 tests pass.

Greptile (@greptileai) review

Comment thread src/cortex/challenges/supervisor.py
A GitHub 403/429/5xx or network error on the attestation lookup is
now a TransientError. The next poll retries it. Only a definitive answer
(no provenance, bad labels, failed canary or rollout) is remembered as a
refusal of that configuration.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@echobt

Copy link
Copy Markdown
Contributor Author

Fixed in b59a14f: attestation outages (403, 429, 5xx or network errors) are now transient and retried on the next poll. Only definitive verdicts are remembered as refusals. The thread is resolved, and 1083 tests pass.

Greptile (@greptileai) review

@echobt
Mathis (echobt) merged commit 6c6d72b into main Sep 24, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant