Upheld

Claims, upheld. | Done means shown.

Harness-agnostic claims-vs-evidence verifier for AI coding agents — matching named claims to deterministic receipts.

Overview

AI coding agents often assert that tasks are complete with phrases like "all 18 tests pass" or "file updated successfully". But asserting is not proving.

Upheld is an independent, harness-agnostic verification tool that audits claims made by AI agents against empirical evidence. It independently re-executes tests deterministically, checks for concrete write and modification evidence (via git status mutations and modification timestamps against --since), flags unacknowledged file mutations, and produces clean audit summaries for developers and CI pipelines.

Quick Start

Upheld is a CLI verifier for any agent harness. Open and DIY loops (OpenCode, Aider, custom shells, CI) get the most leverage: you wire verify into the loop yourself via a skill, stop-hook, or one shell step.

1. Install once (per machine / project)

npm install && npm run build

You do not reinstall every agent turn. (A global npm i -g upheld path comes with a tagged release — not claimable yet.)

2. Write claims

npx . init
# or author .upheld/claims.json / claims.json yourself — see Supported Claim Types

3. Verify

node dist/bin.js verify claims.json
# or: npx . verify claims.json

Optional automation (skill instructions, stop-hook, or CI) shells the same CLI — nothing magic intercepts the model.

Claims vs Evidence Audit Table (Example)

When executed, Upheld evaluates agent claims against on-disk and execution receipts:

Status	Claim Type	Claim Details	Empirical Evidence / Receipt	Verdict
`UPHELD`	`tests_pass`	`cmd: npm test, passed: 18`	Exit code `0`; 18 passed, 0 failed (1.42s)	VERIFIED
`UPHELD`	`file_written`	`path: src/auth/token.ts`	Write evidence verified: modified in git working tree (`M`) & `mtime` >= `--since`	VERIFIED
`UNMET`	`tests_pass`	`cmd: pytest tests/, passed: 5`	Re-run exited with code `1` (3 passed, 2 failed in `test_auth.py:44`)	FAILED
`UNMET`	`file_written`	`path: src/config/env.ts`	No write evidence: file exists on disk but unmodified in git and `mtime` unchanged	FAILED
`UNCLAIMED`	`unclaimed_file`	`(none)`	Unclaimed mutation detected in working tree (`M package-lock.json`)	FLAGGED

Features

Transcript Claim Extraction (Best-effort / Opt-in): Heuristically parses Claude Code and Cursor tool logs (JSONL, JSON, or plain text transcripts) to extract tests_pass and file_written claims. Note: extraction relies on pattern matching and heuristic parsing; it is best-effort and does not guarantee perfect NLP parsing accuracy across arbitrary unstructured text.
Empirical Re-run: Re-runs claimed test commands (pytest, vitest, jest, or arbitrary shell commands) and compares parsed outputs (passed/failed/total counts and exit status) to what was claimed.
File Artifact Verification: Checks for write evidence that claimed file paths were created or modified during the run via git status (M, A, ??) or modification time (mtime) against --since; pre-existing untouched files evaluate to unmet.
Unclaimed Change Detection: Identifies files modified or created in git that were never claimed by the agent.
Self-Contained HTML Reports: Generates dark-theme, single-file, offline HTML reports (--html report.html) with expandable mismatch notes and metric counters with zero external dependencies.
Harness Agnostic: Works with Claude Code, Codex, OpenCode, Cursor, custom CI/CD pipelines, or standalone CLIs.
Report & Strict Modes:
- Report Mode (default): Emits a structured Claims vs Evidence table and exits 0 for transparent observation.
- Strict Mode (--strict): Exits with a non-zero code if any claim is unmet or fails.
SARIF & GitHub Code Scanning: Output standard SARIF 2.1.0 (--format sarif or --sarif) mapping unmet claims to SARIF results for GitHub Code Scanning and VS Code Problems.
GitHub Actions & Job Summary: Automatically renders Markdown job summaries in $GITHUB_STEP_SUMMARY.
Lightweight & Honest: Minimal footprint, zero external bloat, Node 20+.

Documentation

Explore the in-depth documentation in docs/:

Documentation Index
Problem Statement & Motivation: Why autonomous coding agents hallucinate success and how Upheld provides ground truth.
How Verification Works: Architecture, claims schema, test runner parsing, write evidence verification, and unclaimed change detection.
Honesty Rules & Taxonomy: Verification status taxonomy (upheld, unmet, unclaimed) and integrity principles.
Claude Code Hook Integration: Step-by-step guide for Claude Code Stop-hooks and self-correction loops.
CI/CD Integration Guide: Setting up GitHub Actions, GitLab CI, CircleCI, PR comments, and job summaries.
Upheld vs AI PR Reviewers: Why empirical re-execution differs fundamentally from passive diff-reading (CodeRabbit, Copilot).

Supported Claim Types

Upheld evaluates structured claims provided as JSON input.

1. `tests_pass`

Verifies that a specified test command passes by independently executing it and matching exit codes and test counts (passed, failed, total).

{
  "type": "tests_pass",
  "cmd": "npm test -- tests/auth.test.ts",
  "passed": 8,
  "failed": 0,
  "total": 8
}

2. `file_written`

Verifies that files were genuinely written or modified during the run, backed by git working tree mutation signals or modification timestamps within the --since window. Supports single paths, multi-path arrays, and glob pattern expansions with fail-closed honesty.

Single File Path

{
  "type": "file_written",
  "path": "src/services/auth.ts"
}

⚠️ Write Evidence vs. Existence: If a file exists on disk from an earlier run but was never modified during the current session, Upheld marks the claim as UNMET.

Multi-Path List (`paths`)

Verify multiple target artifacts within a single claim. Each path is individually checked for existence and modification evidence:

{
  "type": "file_written",
  "paths": [
    "src/verifier.ts",
    "src/checker.ts",
    "tests/verifier.test.ts"
  ]
}

Glob Pattern Matching (`glob` & `minMatches`)

Verify matched artifacts dynamically across patterns (e.g. **/*.ts). You can specify minMatches (defaults to 1 when glob is used) to enforce a minimum number of matched and modified files:

{
  "type": "file_written",
  "glob": "src/**/*.ts",
  "minMatches": 3
}

Combined Paths & Glob

You can combine explicit paths and glob patterns in a single claim:

{
  "type": "file_written",
  "paths": ["README.md", "package.json"],
  "glob": "tests/**/*.test.ts"
}

Honesty guarantee: If any path in a multi-path claim is missing or unmodified during the evaluation window, or if a glob matches fewer than minMatches, the claim evaluates to UNMET (fail-closed) with itemized per-path evidence reporting.

Full Claims File Example (`claims.json`)

{
  "agent": "vibecoder-v1",
  "task": "fix-auth-expiration",
  "since": "2026-09-05T23:30:00.000Z",
  "claims": [
    {
      "type": "file_written",
      "path": "src/services/auth.ts"
    },
    {
      "type": "tests_pass",
      "cmd": "npm test -- tests/auth.test.ts",
      "passed": 8,
      "failed": 0,
      "total": 8
    }
  ]
}

CLI Options

Usage:
  upheld verify [options] [claims.json]
  cat claims.json | upheld verify [options]

Options:
  --strict             Exit with non-zero code if any claim is unmet (default: exit 0 in report mode)
  --format <type>      Output format: table (default), markdown, or json
  --cwd <path>         Working directory to evaluate claims in (default: current directory)
  --since <timestamp>  Evaluation window start timestamp (ms or ISO date) for file write evidence
  --no-unclaimed       Disable detection of unclaimed modified/untracked files
  --json               Shortcut for --format json
  --markdown           Shortcut for --format markdown
  --summary            Output GitHub Action job summary format
  --summary-file <f>   Append job summary to specified file (or $GITHUB_STEP_SUMMARY)
  -h, --help           Show help message
  -v, --version        Show version

Execution Modes

Report Mode (Default): Inspects empirical evidence, outputs the audit table to stdout and $GITHUB_STEP_SUMMARY if available, and exits with code 0. Ideal for exploratory workflows and agent post-run reviews.
```
node dist/bin.js verify claims.json
```
Strict Mode (--strict): Enforces deterministic integrity by exiting with status 1 whenever any claim is unmet, tests fail, or counts mismatch.
```
node dist/bin.js verify claims.json --strict
```

Bootstrap Configuration (`init`)

Quickly bootstrap an .upheld directory with template claims, a Claude Code Stop-hook script, and a README:

node dist/bin.js init
# or
npx . init

Options:

--github-action: Also create a starter GitHub Action workflow at .github/workflows/upheld.yml.
--force: Overwrite existing files (by default, existing files are skipped).
--cwd <dir>: Target working directory (default: current directory).

Verify Claims in Watch Mode

Watch a claims file for changes and re-verify automatically during development:

node dist/bin.js verify --watch path/to/claims.json
# or
npx . verify -w path/to/claims.json

Extract Claims from Agent Transcripts (Best-effort / Opt-in)

Parse Claude Code or Cursor logs (JSONL, JSON arrays, or text):

# Extract to a claims.json file
node dist/bin.js extract agent-transcript.jsonl --out claims.json
# or
npx . extract agent-transcript.jsonl --out claims.json

# Extract from stdin and pipe directly to verify
cat transcript.jsonl | node dist/bin.js extract | node dist/bin.js verify
# or
cat transcript.jsonl | npx . extract | npx . verify

Verify Claims from a File

node dist/bin.js verify path/to/claims.json
# or
npx . verify path/to/claims.json

Verify Claims via Standard Input

cat claims.json | node dist/bin.js verify
# or
cat claims.json | npx . verify

GitHub Check Run

node dist/bin.js verify --github-check claims.json
# or
npx . verify --github-check claims.json

Generate Self-Contained HTML Reports

node dist/bin.js verify --html report.html claims.json
# or
cat claims.json | node dist/bin.js verify --html report.html

Documentation & Tutorial

Check out our comprehensive step-by-step walkthrough and runnable use case suite:

Tutorial & Runnable Guide: Concrete walkthrough covering test metrics verification, write evidence timestamps, unclaimed side-effect detection, exit code modes, and self-contained fixtures.
Runnable Examples Script: Run bash examples/tutorial/run.sh to execute all verification scenarios locally.

Integrations

Primary path is the harness-agnostic CLI above. Product-specific hooks are optional adapters.

Pre-Commit Hooks (Husky / Lefthook)

Ensure agent and developer claims are upheld before code is committed. See examples/pre-commit/.

Husky (`.husky/pre-commit`)

One-liner:

test -f .upheld/claims.json && npx . verify --strict .upheld/claims.json || true

Lefthook (`lefthook.yml`)

One-liner:

pre-commit:
  commands:
    upheld:
      run: test -f .upheld/claims.json && npx . verify --strict .upheld/claims.json || true

Optional adapter: Claude Code Stop-Hook

Upheld can run as a Claude Code Stop-hook to extract claims from the session transcript and verify them before concluding a session. See examples/claude-code-hook/ for setup and scripts.

Optional adapter: Codex CLI Session Hook

Extract and verify claims from Codex CLI sessions and tool events. See examples/codex-hook/ for adapters and setup.

Optional adapter: OpenCode Session Hook

Normalize tool events from OpenCode sessions into Upheld claims. See examples/opencode-hook/ for adapters and setup.

GitHub Actions

To verify claims in CI, run the built CLI directly or invoke local verify steps:

- name: Verify Agent Claims
  run: |
    npm ci && npm run build
    node dist/bin.js verify .upheld/claims.json

(Note: Direct composite action usage via uses: chuofringer/upheld@main will be available after PR #1 merges to main.)

Why Upheld?

Most coding agent harnesses rely either on:

Self-reported completion: The agent declares "I ran the tests and they passed", leading to false positives, silent omissions, and hallucinated success.
Heavyweight finish-review blockers: Rigid, monolithic review gates that freeze agent workflows or demand proprietary orchestrators.

Upheld provides a lightweight, focused wedge: deterministic receipt verification. It doesn't care how the agent was prompted or what model produced the code — it only verifies whether the agent's explicit claims match verifiable, reproducible facts on disk and in execution.

False-Completion Corpus

Upheld includes a curated fixture corpus of common false-completion patterns exhibited by coding agents (skipped tests, phantom file writes, inflated pass counts, swallowed exit codes).

See examples/corpus/ and run the corpus validation suite:

npm run corpus

Publishing (Maintainers Only / Dry-Run Verification)

Notice: Publishing to npm is restricted to repository owners / maintainers (@chuofringer) and is planned for a future release. Upheld is not yet published to npm.

To verify the package contents that will be included in future releases via dry-run:

npm pack --dry-run

When ready for release (maintainers only):

npm publish --access public --dry-run
# Actual publish (owners only upon release):
# npm publish --access public

License

Distributed under the MIT License. See LICENSE for details.

_{Built by vibemapper · Claims, upheld.}

Name		Name	Last commit message	Last commit date
.github		.github
docs		docs
examples		examples
site		site
src		src
tests		tests
.gitignore		.gitignore
.npmignore		.npmignore
CHANGELOG.md		CHANGELOG.md
LICENSE		LICENSE
README.md		README.md
action.yml		action.yml
package-lock.json		package-lock.json
package.json		package.json
tsconfig.json		tsconfig.json
vitest.config.ts		vitest.config.ts

Latest commit

History

Folders and files

Repository files navigation

Upheld

Overview

Quick Start

1. Install once (per machine / project)

2. Write claims

3. Verify

Claims vs Evidence Audit Table (Example)

Features

Documentation

Supported Claim Types

1. tests_pass

2. file_written

Single File Path

Multi-Path List (paths)

Glob Pattern Matching (glob & minMatches)

Combined Paths & Glob

Full Claims File Example (claims.json)

CLI Options

Execution Modes

Bootstrap Configuration (init)

Verify Claims in Watch Mode

Extract Claims from Agent Transcripts (Best-effort / Opt-in)

Verify Claims from a File

Verify Claims via Standard Input

GitHub Check Run

Generate Self-Contained HTML Reports

Documentation & Tutorial

Integrations

Pre-Commit Hooks (Husky / Lefthook)

Husky (.husky/pre-commit)

Lefthook (lefthook.yml)

Optional adapter: Claude Code Stop-Hook

Optional adapter: Codex CLI Session Hook

Optional adapter: OpenCode Session Hook

GitHub Actions

Why Upheld?

False-Completion Corpus

Publishing (Maintainers Only / Dry-Run Verification)

License

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages