Skip to content

fix: restore Forge validation and classify invalid CLI JSON - #103

Open
nicholas-ruest wants to merge 3 commits into
mainfrom
metaharness/forge-security-recovery-20261013
Open

nicholas-ruest wants to merge 3 commits into
mainfrom
metaharness/forge-security-recovery-20261013

Conversation

@nicholas-ruest

@nicholas-ruest nicholas-ruest commented Oct 14, 2026 •

Copy link
Copy Markdown
Member

Summary

Resumes the existing security-recovery branch; keeps its dependency cleanup and deterministic contract-test repair.

This cycle fixes CLI generator request validation: null/primitive/array JSON and untrusted metadata no longer trigger coercion exceptions; schema failures have a typed failureType and return HTTP400 instead of guessing status from error text. Internal failures retain HTTP500. Existing service persistence, Rust, IAM and deployment behavior are unchanged.

Current measured results

Baseline 733f97caefd62ca750e69ba1b24153363a836e19; source db68b9c6904c8566466cd8eeda3842fe4eaa88e9; final ac170c4d0a18e5fd5266a08d3acb38c882bdf574.

Gate Baseline Candidate
Native loopback HTTP invalid-input cases 0/7 7/7
Compatibility HTTP anchors 4/4 4/4
Existing/new unit assertions 1070 passing 1077 passing, original1070 retained
Build pass pass
Strict TypeScript diagnostics 131 129 (still failed)
ESLint errors 1421 1421 (still failed)
Files failing format 75 74
Clean-tree coverage lines/statements 70.11% 70.14% (90% threshold unchanged)
Coverage functions / branches 86.40% / 80.08% 86.42% / 80.16%
Production npm audit dependency lock unchanged 0 findings
Full dependency npm audit dependency lock unchanged 26 findings:4critical,18high,4moderate

These are component validation results, not permission to merge. The HTTP fixture exercises the real exported edge handler over loopback TCP; it is not a Cloud Run/IAM/RuVector-delivery test. Existing default in-memory decision-emitter behavior was not replaced or presented as real persistence.

Real tool execution

Pinned Ruflo3.56.3 + Transformers4.3.1, MetaHarness0.4.17, Darwin0.10.3, Flywheel0.1.12. Current upstream documentation revision e0dfd44da72b7adfb7c57fdd42d124be58ed7086.

  • Ruflo swarm swarm-1792281574110-6kd88m, task task-mvd269ml; one coordinator, no delegated workers.
  • Offline post-task used fail-closed seccomp + checksum-pinned real MiniLM384 embeddings, revision 751bff37182d3f1213fa05d7196b954e230abad9. learningUpdates.available/outcomePersisted, feedback.recorded, trajectory.recorded all true. Overall task correctly recorded unsuccessful because global gates failed.
  • MetaHarness score82→82, hard constraints6/6; static estimate, not compiler proof.
  • Darwin real sandbox, seed2026101720, one generation/two children/concurrency1/Pareto: all0.985; scorePolicy/retryPolicy variants rejected, NO_IMPROVEMENT. Nine actual npm test invocations exited0.
  • Flywheel frozen seven-case holdout0→1, six anchors1→1, component promotion only; signed replay8/8 and inverse parent bytes retained. Gate fingerprint c5942e7c9bf8f28fe7af47a6ec3887d11eeb3f8c8e8601cc6a7a54969ea0b981.
  • Replay independently verified from the committed compressed artifact on final SHA.

Merge blockers

Do not merge. Types/lint/format/coverage and full dependency security are red. RuOS bootstrap passed and OpenSSL3.0.13 is verified; real locked Rust invocation exits101 at the existing invalid benchmark manifest (missing benches/forge_benchmarks.rs), before any Rust test executes. No missing-OpenSSL or successful-Rust claim is made.
Hosted source validation 39081386440: Node20/21 tests and build pass; type/lint/format/coverage/aggregate fail. Source security fails full audit and Dependency Review (GitHub reports dependency graph unavailable); CodeQL, static security and secret scan pass. A green Security Summary alone is insufficient.

Final validation 39081656729 and security 39081656740 are checked separately before any merge decision.

Evidence / reproducibility

  • Full coverage/deferred ledger and exact commands: all58nonarchived repositories, both repository pages, all117openPRs,57implementation deferrals.
  • Frozen acceptance contract
  • Flywheel summary, full signed flywheel-results.json.gz, raw darwin-runs.tar.gz alongside.
  • Run node --import tsx tests/service/verify-cli-input.mjs ..
  • Run node tests/evaluate-cli-input.mjs BASELINE CANDIDATE PINNED_TOOLS OUTPUT.json with installed pinned tools.
  • No thresholds, exclusions, test conditions or scoring rules changed. Test utilities placed under existing tests directory; generated local Darwin files excluded from publication, not by new coverage exclusions.

Conflicts and rollback

Main remains 4b5e43618b8fd1b8f63bddd24ff524510d9514bc; conflict-free at inspection. Only expected-head fast-forwards, no force push or protection bypass.
Revert this cycle's commits db68b9c6904c8566466cd8eeda3842fe4eaa88e9 and ac170c4d0a18e5fd5266a08d3acb38c882bdf574 to retain the earlier dependency recovery. Signed inverse restores exact parent handler bytes; reverse patch applicability verified. To undo the entire PR after a future merge, revert its merge commit. No persistence/schema rollback is required.

Exact final commit verification

Final head ac170c4d0a18e5fd5266a08d3acb38c882bdf574 was revalidated after publication in clean local Node24 and RuOS Node20.20.2/npm10.8.2 checkouts; tracked files remained clean.

  • npm ci --ignore-scripts, npm test, npm run build, native HTTP fixture: exit0;1077/1077 tests.
  • npm run type-check:2; npm run lint:1; npm run format:check:1; npm run test:coverage:1.
  • RuOS and hosted final coverage:70.11%lines/statements,86.91%functions,80.16%branches. Clean Node24 controlled baseline→candidate coverage70.11→70.14%; do not compare cross-runtime function percentages as a performance claim.
  • Exact final cargo test --manifest-path forge-benchmarks/Cargo.toml --locked:101, unchanged missing bench path; no Rust tests executed.
  • Signed compressed Flywheel replay independently verifies8/8 on local Node24 and RuOS Node20.
  • Final validation39081656729 completed failure: Node20/21 tests and build success; type/lint/format/coverage/Quality Gate failed.
  • Final security39081656740 completed failure: dependency scan/review failed; CodeQL/static/secret/license passed. Security Summary success does NOT override those failures.
  • Final performance39081656751 completed failure: benchmark job passed; baseline comparison failed, regression detection skipped. No speedup or regression verdict inferred.
  • Main rechecked 4b5e43618b8fd1b8f63bddd24ff524510d9514bc; PR remains open, non-draft, mergeable/no conflicts, NOT MERGED. No approvals/protections bypassed.

Final failure diagnosis: performance comparison exits1 because main does not track benchmark-results/latest.json (git pathspec failure). Dependency Review exits1 because GitHub reports dependency graph/review unavailable for this repository. No repository administration setting was changed, and neither failure was bypassed.

@github-actions

Copy link
Copy Markdown

🔒 Security Scan Results

Scan Type Status
Dependency Scan ⚠️ failure
CodeQL Analysis ✅ success
Secret Scan ✅ success
License Check ✅ success
SAST ✅ success

⚠️ Some security scans have warnings or failed. Please review the details.


Automated security scanning by GitHub Actions

@github-actions

Copy link
Copy Markdown

🔒 Security Scan Results

Scan Type Status
Dependency Scan ⚠️ failure
CodeQL Analysis ✅ success
Secret Scan ✅ success
License Check ✅ success
SAST ✅ success

⚠️ Some security scans have warnings or failed. Please review the details.


Automated security scanning by GitHub Actions

@nicholas-ruest nicholas-ruest changed the title fix(security): restore deterministic Forge validation fix: restore Forge validation and classify invalid CLI JSON Oct 18, 2026
@github-actions

Copy link
Copy Markdown

🔒 Security Scan Results

Scan Type Status
Dependency Scan ⚠️ failure
CodeQL Analysis ✅ success
Secret Scan ✅ success
License Check ✅ success
SAST ✅ success

⚠️ Some security scans have warnings or failed. Please review the details.


Automated security scanning by GitHub Actions

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant