Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,5 +47,6 @@ Every reproduced race requires a failing-before and passing-after regression. As
- In evidence records, label cited commits as baselines, intermediate checkpoints, or validated heads. Keep final exact-head results in a place that can name the resulting commit, such as the PR body or CI record.
- Reproduce summary-only review concerns or turn them into a concrete follow-up issue. Do not repeatedly patch vague wording without a failure case.
- A validation fixture for a summary-only concern must assert the disputed intermediate representation or state before using a downstream outcome as evidence that the concern was exercised.
- Manual assignment evidence must include a reproducible per-segment manifest with stable choice ID, path, operation, assigned item, and final scope; aggregate counts alone are insufficient.
- For each review round, record what changed, what was declined and why, and the regression evidence. Re-request review until a round returns no new findings.
- Treat review-lesson extraction as a merge gate. Before invoking merge, classify every review finding in the PR body as: covered by an existing rule (cite it), captured by a new rule in this branch (cite it), or one-off (record why). Do not merge until this audit is complete and every required `AGENTS.md` update is included in the reviewed head. Omit rules that merely repeat existing guidance.
18 changes: 18 additions & 0 deletions docs/experiments/manual-assignment.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
{
"base": "5a2685cc709ca8f9f384e35e4d733012c7290d69",
"head": "442147d1bc9a33edeca1a605c9de3ada13f397f1",
"segments": [
{ "choice_id": "96c76cee6518b177f272d8dba1f8db83314d0c09b62f036c6d6f41de40227d5c", "kind": "text", "operation": "+", "path": "AGENTS.md", "before_row": "Unplanned", "item": "P4", "scope": "in-scope" },
{ "choice_id": "6000e022cb5f2c7fb602ba9dbd6b590cdf0672b1bdc984c036bf563423a86afd", "kind": "text", "operation": "+", "path": "AGENTS.md", "before_row": "Unplanned", "item": "P4", "scope": "in-scope" },
{ "choice_id": "aa5dbe642c11d5f5f6f1630c0f979006b8e0e3182da3e0aeee8485fa997784d1", "kind": "file", "operation": null, "path": "docs/experiments/review-summary-edge-cases.md", "before_row": "Unplanned", "item": "P3", "scope": "in-scope" },
{ "choice_id": "dfcf7097d83b6d55df0e4959f111370a96976b1c25f327e9d51838fc8224ceaf", "kind": "text", "operation": "+", "path": "docs/experiments/review-summary-edge-cases.md", "before_row": "Unplanned", "item": "P3", "scope": "in-scope" },
{ "choice_id": "7a5227f9a6be94d6c16d1ea058ae3a5d956a5b53f93333a15d3b778222a90218", "kind": "text", "operation": "-", "path": "test/plant.test.ts", "before_row": "Unplanned", "item": "P2", "scope": "in-scope" },
{ "choice_id": "142cf07bee22dbe7cf281b0d97efa49f9a502fed62701de8ec0c94500a94c788", "kind": "text", "operation": "+", "path": "test/plant.test.ts", "before_row": "Unplanned", "item": "P2", "scope": "in-scope" },
{ "choice_id": "36cb67aa87587c9159f19ce0bac78a4d67f1348f55108f99e1fed99e5273a414", "kind": "text", "operation": "+", "path": "test/plant.test.ts", "before_row": "Unplanned", "item": "P2", "scope": "in-scope" },
{ "choice_id": "88fada2e5b9a307f6db4fa6ccdbbf9e5bfbf00f26cbcacb33ac5bc141d74b85e", "kind": "text", "operation": "+", "path": "test/plant.test.ts", "before_row": "Unplanned", "item": "P2", "scope": "in-scope" },
{ "choice_id": "9f010f7ae66fd80d72c5d78d45d4effbcdac76ebb6dfadc4efd4e66ac4af9f86", "kind": "text", "operation": "-", "path": "test/review.test.ts", "before_row": "Unplanned", "item": "P1", "scope": "in-scope" },
{ "choice_id": "987cbec380909ee199bbe4df04a7d700c6f2a1f284c52f08120c159f8e92c35d", "kind": "text", "operation": "+", "path": "test/review.test.ts", "before_row": "Unplanned", "item": "P1", "scope": "in-scope" },
{ "choice_id": "6b276597aa7294f44e5a549e38d423876266fe6a9a502bb5be089db2223a8976", "kind": "text", "operation": "+", "path": "test/review.test.ts", "before_row": "Unplanned", "item": "P1", "scope": "in-scope" },
{ "choice_id": "c9379939cb69513a37cbb385440217283975294a2f8b340839d5f9d09ea126df", "kind": "text", "operation": "+", "path": "test/review.test.ts", "before_row": "Unplanned", "item": "P1", "scope": "in-scope" }
]
}
19 changes: 19 additions & 0 deletions docs/experiments/manual-assignment.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# Manual real-issue assignment

This prerequisite exercise uses real issue [#10](https://github.com/codeabovelab/codeboost/issues/10) and merged [PR #17](https://github.com/codeabovelab/codeboost/pull/17). It is an attribution exercise, not a timed trial and not evidence for the go/no-go thresholds.

- Base: `5a2685cc709ca8f9f384e35e4d733012c7290d69`
- Head: `442147d1bc9a33edeca1a605c9de3ada13f397f1`
- Ledger setup: the squash commit was intentionally recorded as foreign so every resulting segment required an explicit manual assignment.
- Plan: P1 rename-reassignment regression, P2 literal-path regression, P3 evidence record, and P4 durable review rules.

| Plan item | Declared file | Assigned segments | Final attribution | Final scope |
|---|---|---:|---|---|
| P1 | `test/review.test.ts` | 4 text | Attributed | In scope |
| P2 | `test/plant.test.ts` | 4 text | Attributed | In scope |
| P3 | `docs/experiments/review-summary-edge-cases.md` | 1 file + 1 text | Attributed | In scope |
| P4 | `AGENTS.md` | 2 text | Attributed | In scope |

Codeboost exposed 12 segments before assignment, all in the Unplanned row. Each segment mapped to exactly one item through that item's declared file contract. After the assignments, the per-item segment counts were 4, 4, 2, and 2; all four rows reported `✓ Attributed` and `✓ In scope`. No segment was unclear or multiply owned, so this exercise produced no ambiguity fixture. The earlier rename and literal-path concerns already have focused fixtures in `test/review.test.ts` and `test/plant.test.ts`.

The assignment was rerun from a detached worktree at the exact head using a fresh SQLite review store. The stored choices, rather than a hand-written diff classification alone, produced the final attribution and scope results above. `manual-assignment.json` records every stable choice ID with its kind, operation, path, original row, assigned item, and final scope so the 12 assignments can be checked individually.
24 changes: 16 additions & 8 deletions docs/experiments/review-protocol.md
Original file line number Diff line number Diff line change
@@ -1,19 +1,27 @@
# Plan-indexed review: go/no-go protocol

Status: **not started; issue pairs and human reviewer still required**. This document does not claim a passed gate. Complete the pair table and commit its exact revision before any timed review. No pair or threshold may change once the first review begins.
Status: **pair selection frozen; all four pairs prepared; reviews not started**. Max Hwang (`mchwang`) is the blinded reviewer and Codex is the experiment operator. This document does not claim a passed gate. Every row's PR and exact base/head SHAs are committed below. No issue, assignment, order, or threshold may change once the first review begins.

## Pair selection and order

Choose four pairs of comparable, small real issues in a repository owned by the reviewer. Match each pair on scope, language, and estimated review complexity before implementing either issue. Record issue URLs, PR URLs, base/head SHAs, and which review method each gets. Do not reuse the demo fixture as an experimental issue.

| Pair | Real issue A / PR | Real issue B / PR | A method | B method |
All eight issues are in `codeabovelab/guardyx-gstack`. A/B placement within each pair was randomized once before implementation; the method pattern remains the alternating pattern approved in the protocol.

| Pair | Real issue A / PR / base..head | Real issue B / PR / base..head | A method | B method |
|---|---|---|---|---|
| 1 | Pending selection | Pending selection | Raw GitHub diff | codeboost |
| 2 | Pending selection | Pending selection | codeboost | Raw GitHub diff |
| 3 | Pending selection | Pending selection | Raw GitHub diff | codeboost |
| 4 | Pending selection | Pending selection | codeboost | Raw GitHub diff |
| 1 — companion documentation | [#493](https://github.com/codeabovelab/guardyx-gstack/issues/493) / [PR #600](https://github.com/codeabovelab/guardyx-gstack/pull/600) / `f95a7ebaaa1e48a81fa724e8d6be677a17e7ee69..6a273dab42005e01a6f4c2ea92297f40ad29e0d7` | [#176](https://github.com/codeabovelab/guardyx-gstack/issues/176) / [PR #601](https://github.com/codeabovelab/guardyx-gstack/pull/601) / `f95a7ebaaa1e48a81fa724e8d6be677a17e7ee69..f61a793ebe4e7352f7ebf74238adfad3c18750d0` | Raw GitHub diff | codeboost |
| 2 — agent-key security | [#506](https://github.com/codeabovelab/guardyx-gstack/issues/506) / [PR #602](https://github.com/codeabovelab/guardyx-gstack/pull/602) / `f95a7ebaaa1e48a81fa724e8d6be677a17e7ee69..11a5d74d9bcdd0dcd1faa96e02cfe101ed9c8b6a` | [#443](https://github.com/codeabovelab/guardyx-gstack/issues/443) / [PR #603](https://github.com/codeabovelab/guardyx-gstack/pull/603) / `f95a7ebaaa1e48a81fa724e8d6be677a17e7ee69..042999d9ebc3dc61fb029b21644bd20b4537a347` | codeboost | Raw GitHub diff |
Comment thread
mchwang marked this conversation as resolved.
| 3 — asynchronous notifications | [#13](https://github.com/codeabovelab/guardyx-gstack/issues/13) / [PR #608](https://github.com/codeabovelab/guardyx-gstack/pull/608) / `f95a7ebaaa1e48a81fa724e8d6be677a17e7ee69..78f62ee413a291a5f0ea555c39bc8d61b6721f9b` | [#429](https://github.com/codeabovelab/guardyx-gstack/issues/429) / [PR #609](https://github.com/codeabovelab/guardyx-gstack/pull/609) / `f95a7ebaaa1e48a81fa724e8d6be677a17e7ee69..b8cd873ac688fa27ac841fb2e2df2d6d135446c6` | Raw GitHub diff | codeboost |
| 4 — compliance evidence | [#504](https://github.com/codeabovelab/guardyx-gstack/issues/504) / [PR #610](https://github.com/codeabovelab/guardyx-gstack/pull/610) / `f95a7ebaaa1e48a81fa724e8d6be677a17e7ee69..4ce1259c396ed40001efc02216a4cd181f8e50b4` | [#507](https://github.com/codeabovelab/guardyx-gstack/issues/507) / [PR #611](https://github.com/codeabovelab/guardyx-gstack/pull/611) / `f95a7ebaaa1e48a81fa724e8d6be677a17e7ee69..f9988770b89a14f006adf0ecefb15115d3d748d0` | codeboost | Raw GitHub diff |

The pairs are matched before implementation on repository, primary implementation language, scope, and expected review surface. Pair 1 changes operational/compliance documentation without runtime behavior; pair 2 changes agent-key lifecycle across the API and UI; pair 3 adds medium asynchronous notification delivery across backend jobs and consumer-facing interfaces; pair 4 adds compliance evidence across stored invocation data, exports, mapping data, and PDF surfaces. If implementation discovery makes a pair materially unequal or blocked, stop before the first timed review and record a new frozen protocol revision rather than silently substituting an issue.

Selection history: the first frozen draft paired #554 with #556. Before either implementation began, their required baseline check found that both depend on live 1,400-action/session telemetry unavailable from the operator's environment. A proposed replacement included #335, but inspect-before-build then proved that `main` already contains its requested in-flight guard even though the issue remains open. A current-tree audit also excluded other stale open issues whose requested work has shipped. The table above contains only work confirmed absent from current `main`; every A/B placement was randomized after that audit. No timed review, experimental implementation, or plant had started before this final selection.

Before timed review began, an automated reviewer was mistakenly requested on the original pair 3 and pair 4 draft PRs. That invalidated reviewer blindness for those four packages. PRs #604–#607 were closed, fresh randomized packages were created without changing their issues or method assignments, and the replacement PRs and exact revisions are recorded above. Automated review stays off every experimental draft until its human decision is recorded.

The reviewer must not implement the paired changes or inspect the plants before deciding on the PR. An operator prepares plans and commits by hand, records trusted ledger ownership, and supplies both the plan and code to each review method. The earlier one-issue manual assignment also remains required: record how every change maps to the attribution table and turn unclear cases into fixtures.
The reviewer must not implement the paired changes or inspect the plants before deciding on the PR. The operator must not expose plant inputs, `sealed.json`, mappings, planted locations, or unblinded implementation notes in the PR, review package, task messages, or protocol updates. An operator prepares plans and commits by hand, records trusted ledger ownership, and supplies both the plan and code to each review method. The earlier one-issue manual assignment is recorded in `manual-assignment.md`; all 12 segments were assigned and in scope, with no unclear case requiring another fixture.
Comment thread
mchwang marked this conversation as resolved.

## Planting

Expand All @@ -37,4 +45,4 @@ Report declared-file catch rate for both methods without a pass threshold. Stop

## Current evidence

Engineering tests prove browser interactions, persistence, attribution, and planting mechanics. They do **not** establish human review speed or catch rates. No real issue pairs, timed decisions, or catch-rate results exist yet.
Engineering tests prove browser interactions, persistence, attribution, and planting mechanics. The separate manual assignment proves that one real merged issue can be completely classified through stored review choices. Neither establishes human review speed or catch rates. All four pairs are prepared at the eight exact PR revisions above, and every revision passed the repository preflight. No timed decision or catch-rate result exists yet.
2 changes: 1 addition & 1 deletion docs/experiments/review-summary-edge-cases.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,4 +22,4 @@ Result: no defect reproduced. The summary supplied no expected color, state toke

## Gate result

All three concerns are resolved as validated behavior. No issue supplied a failing case that justified a production patch. Issue #3's manual assignment and paired human go/no-go experiment remain pending and must use the frozen protocol in `review-protocol.md`.
All three concerns are resolved as validated behavior. No issue supplied a failing case that justified a production patch. Issue #3's manual assignment is recorded in `manual-assignment.md`; its paired human go/no-go experiment remains pending and must use the frozen protocol in `review-protocol.md`.
2 changes: 1 addition & 1 deletion docs/implementation/build-step-1.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ Fixed aggregate diff retention outside the blob budget. The adapter limits total

This is a working foundation, not a completed application or a claim that all implementation tasks are done. T18's pure validation/edit core is present; its agent adapters, import UI, and persistence are pending. Ledger storage, rebase mappings, and the read-only review screen remain next. The already-fixed GitHub check belongs to the later GitHub/runner integration.

The design's manual real-issue assignment and timed go/no-go experiment have not been performed. Disposable Git histories are engineering tests, not evidence that plan-indexed review beats raw review. Write and commit the experiment protocol before using the real review screen for that comparison. Do not proceed to merging, agent execution, planning UI, queue, or learning until the documented gate passes.
The design's manual real-issue assignment is complete, and the timed go/no-go experiment is prepared but has not been performed. Disposable Git histories and the assignment exercise are engineering evidence, not evidence that plan-indexed review beats raw review. Use the frozen experiment protocol for the timed comparison. Do not proceed to merging, agent execution, planning UI, queue, or learning until the documented gate passes.

## Alignment with the merged v1 contract (#6)

Expand Down
2 changes: 1 addition & 1 deletion docs/implementation/read-only-review.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ Store schema v2 adds the review counter and per-item notes through a transaction

- No agent answers, test execution, AI findings ingestion, send-to-agent action, or merge control is included. These belong after the go/no-go gate. Four checks distinguish unavailable evidence from success.
- File cards show mode/path/object IDs and blob byte sizes. PNG/JPEG/GIF/WebP previews are bounded to 1 MiB each and 4 MiB across a history; unsupported/oversized images say unavailable. Gitlink byte sizes are not applicable. SVG/HTML is never embedded.
- The protocol at `docs/experiments/review-protocol.md` must be filled with the real pairs and committed before the first timed review. The manual assignment, real paired PRs, human timing, and final result are pending. Do not mark issue #3 closed or claim the gate passed.
- The real issue pairs are frozen in `docs/experiments/review-protocol.md`, and the completed manual assignment is recorded in `docs/experiments/manual-assignment.md`. All four pairs are prepared at their exact PR revisions; human timing and the final result remain pending. Do not mark issue #3 closed or claim the gate passed.
- The planting helper is intentionally limited to disposable clones and supported regular top-level paths; it never publishes PRs.

## Validation
Expand Down
Loading