Skip to content

test(simulation): long-run determinism suite and golden hashes (Phase 2, PR 5/5) - #90

Merged
CodeByBryant merged 1 commit into
mainfrom
test/deterministic-replay
Sep 27, 2026
Merged

CodeByBryant merged 1 commit into
mainfrom
test/deterministic-replay

Conversation

@CodeByBryant

Copy link
Copy Markdown
Owner

Summary

Phase 2, PR 5 of 5 (the last one). Makes the exit criteria permanent, ongoing checks rather than one-off manual verification: 100,000-tick runs, golden hashes, and a cross-platform (ubuntu + windows) CI job. This closes out Phase 2.

  • tests/determinism/long-run.test.ts: 100k ticks with periodic validate(); same-seed -> identical hash and snapshot at 100k; different seed -> different hash; run(37412) + run(62588) == run(100000); 100 chunks of 1000 == one run(100000); single-step() loop == run(); extinction reached then never reversed across 99,000 further ticks.
  • tests/golden/state-hashes.json + tests/determinism/golden.test.ts: six pinned scenarios (default, the two roadmap scenarios, verbose event detail, zero population, a 100k-tick run), each reproduced from scratch and compared to a hard-coded hash. The test checks hashSchemaVersion against HASH_SCHEMA_VERSION before anything else, so a layout change fails with an explicit "bump and regenerate" message rather than 100+ opaque mismatches.
  • Kept out of the required path: a package-local vitest.config.ts excludes tests/determinism/** from the default pnpm test; pnpm test:determinism runs it explicitly. New CI job Determinism, matrix [ubuntu-latest, windows-latest], continue-on-error: true, not in any needs: chain — never blocks a merge.
  • tests/stream-usage.test.ts (regular, fast suite): a static scan asserting systems/ and world/ never call .get('genetics'|'reproduction'|'learning'|'events'). This is deliberately not a hash-based check — a system silently pulling from a reserved stream would still be perfectly reproducible (same seed, same result), so the determinism tests alone can't catch that "isolation" regression; this closes that gap the roadmap's testing checklist asks for.
  • Split packages/simulation's tsconfig (tsconfig.json now src-only; new tsconfig.test.json adds @types/node for tests/) so the new tests can use fs/path/url, without weakening src/'s no-Node/no-browser guarantee — verified with a negative check (process.env in src/ still fails typecheck).

Simulation impact

  • No simulation behavior changed (tests, golden data, and CI only)

Determinism

  • Determinism tests were added
  • Not applicable otherwise: no engine code changed

Testing

Fresh clone of the pushed branch: install --frozen-lockfile, format:check, check (191 fast tests, unchanged from before this PR), build, and pnpm test:determinism (15 tests, ~53s) all pass.

  • Negative checks (temporary, reverted): bumping the golden file's hashSchemaVersion fails with the intended message; doubling the metabolism cost formula in a real system file fails 5 of 6 golden entries (the 6th has zero organisms, so it's correctly insensitive) — proof the suite is a real regression guard, not a tautology; process.env added to src/index.ts still fails tsconfig.json's typecheck after the tsconfig split.
  • The default 5s Vitest timeout was too short for tests running multiple 100k-tick simulations back to back; vitest.determinism.config.ts sets testTimeout: 60_000.

Review notes

  • Six golden scenarios felt like reasonable coverage (default, both roadmap scenarios, verbose events, zero population, one full 100k run) without the file becoming unwieldy; happy to add more.
  • The Determinism job's cross-platform value is real but unverified by me beyond typical Node/V8 behavior being OS-independent — this PR is what first puts it in front of actual Windows CI runners.

This closes Phase 2: a seeded world runs 100,000 ticks headlessly, same seed -> same hash, different seed -> different hash, no NaN, food/starvation/death events work, no browser/Node dependency in the engine, and math/time/randomness/events all have unit tests.

🤖 Generated with Claude Code

Adds tests/determinism/ (100k-tick runs, chunked-replay equivalence at
scale, extinction permanence over a long remaining run) and
tests/golden/state-hashes.json (six pinned scenarios, checked against
HASH_SCHEMA_VERSION first). Kept out of the default pnpm test via a
package-local vitest.config.ts exclude; run explicitly with
pnpm test:determinism, wired into a new non-required, non-blocking
Determinism CI job on ubuntu and windows.

Also adds tests/stream-usage.test.ts (fast, regular suite): a static
scan proving the Phase 2 engine never reads the genetics/reproduction/
learning/events streams, complementing the hash-based determinism
tests, which cannot by themselves catch an unused-stream regression.

Splits packages/simulation's tsconfig so tests/ may use Node builtins
(fs/path/url) while src/ keeps its no-Node/no-browser restriction,
verified with a negative check.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings September 27, 2026 01:24

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@CodeByBryant
CodeByBryant merged commit a64e852 into main Sep 27, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants