Repository navigation
Proposal: optional audit manifest for reproducible MLCube runs #367
Description
Proposal
Would MLCube be open to an optional run audit manifest for MLCube executions?
MLCube already focuses on portability and reproducibility. A small sidecar manifest could make benchmark runs easier to review, compare, cite, and publish safely without changing the core MLCube task interface.
Suggested manifest shape
{
"schema_version": "mlcube.run_audit.v1",
"mlcube_task": "run",
"runner": "docker",
"image": "...",
"mlcube_config_hash": "...",
"benchmark": "...",
"dataset_refs": [
{
"source_id": "...",
"kind": "dataset",
"provenance": "...",
"redaction_status": "safe_for_public_log"
}
],
"result_paths": ["..."],
"provenance": {
"repo": "...",
"commit": "...",
"created_at": "..."
},
"claim_status": "diagnostic",
"redaction_status": "safe_for_public_log"
}Why this may help
- makes it clearer which MLCube/config/image produced a benchmark result
- preserves provenance such as runner, image, config hash, repo commit, dataset refs, and result paths
- separates diagnostic/internal runs from results intended for public reports or model cards
- gives downstream benchmark users a standard place for audit-safe metadata
- avoids storing raw secrets, private paths, tokens, or full sensitive arguments in public logs
Scope I would keep small
If maintainers think this is useful, I can prepare a follow-up PR that:
- documents an optional manifest schema
- adds a minimal example manifest under docs/examples or docs/getting-started
- keeps the manifest opt-in and backward compatible
- does not change existing runner behavior by default
- does not add external service dependencies
This is motivated by work in the AANA project around audit-safe AI evaluation artifacts, but the contribution would be generic to MLCube and would not require AANA as a dependency.
Activity
HarperZ9 commentedon Jul 2, 2026
I like keeping this optional and sidecar-based. One refinement that may make it easier to implement without changing the runner contract is splitting the shape into a local
run_receiptand a shareablepublic_attestation.A minimal receipt could be:
{ "schema": "mlcube.run_receipt.v1", "run_id": "...", "created_at": "...", "mlcube": { "task": "run", "mlcube_yaml_hash": "sha256:...", "parameters_hash": "sha256:...", "redacted_parameter_keys": ["..."] }, "runtime": { "runner": "docker|singularity|host", "image_ref": "...", "image_digest": "sha256:...", "container_runtime_version": "...", "os": "...", "python": "...", "cuda": "...", "package_lock_hash": "sha256:...", "env_hash": "sha256:..." }, "code": { "repo": "...", "commit": "...", "dirty_tree": false }, "data": [ { "dataset_id": "...", "dataset_version": "...", "manifest_hash": "sha256:...", "access_policy_ref": "..." } ], "outputs": [ { "path": "metrics.json", "kind": "metrics|model|log|artifact", "hash": "sha256:..." } ], "execution": { "started_at": "...", "finished_at": "...", "exit_code": 0 }, "redaction_profile": "public-log-safe", "claim_status": "diagnostic|candidate|published" }Then
public_attestationcan be a strict subset/digest-only projection of the receipt for papers, model cards, benchmark dashboards, or GitHub issues. That keeps private paths, tokens, raw arguments, and restricted dataset details out of public artifacts while still making the run auditable.Suggested CI tests for a first PR:
--audit-manifest out.jsonmlcube.yamlhash, dataset manifest hash, and output hashes are present for the sample runThat pattern maps well to Project Telos/Crucible-style receipts: keep the private source payload local, publish hashes/refs/verdicts, and make claims re-checkable without turning the manifest into another data leak surface.