test(simulation): long-run determinism suite and golden hashes (Phase 2, PR 5/5) - #90
Merged
Merged
Conversation
Adds tests/determinism/ (100k-tick runs, chunked-replay equivalence at scale, extinction permanence over a long remaining run) and tests/golden/state-hashes.json (six pinned scenarios, checked against HASH_SCHEMA_VERSION first). Kept out of the default pnpm test via a package-local vitest.config.ts exclude; run explicitly with pnpm test:determinism, wired into a new non-required, non-blocking Determinism CI job on ubuntu and windows. Also adds tests/stream-usage.test.ts (fast, regular suite): a static scan proving the Phase 2 engine never reads the genetics/reproduction/ learning/events streams, complementing the hash-based determinism tests, which cannot by themselves catch an unused-stream regression. Splits packages/simulation's tsconfig so tests/ may use Node builtins (fs/path/url) while src/ keeps its no-Node/no-browser restriction, verified with a negative check. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase 2, PR 5 of 5 (the last one). Makes the exit criteria permanent, ongoing checks rather than one-off manual verification: 100,000-tick runs, golden hashes, and a cross-platform (ubuntu + windows) CI job. This closes out Phase 2.
tests/determinism/long-run.test.ts: 100k ticks with periodicvalidate(); same-seed -> identical hash and snapshot at 100k; different seed -> different hash;run(37412) + run(62588)==run(100000); 100 chunks of 1000 == onerun(100000); single-step()loop ==run(); extinction reached then never reversed across 99,000 further ticks.tests/golden/state-hashes.json+tests/determinism/golden.test.ts: six pinned scenarios (default, the two roadmap scenarios, verbose event detail, zero population, a 100k-tick run), each reproduced from scratch and compared to a hard-coded hash. The test checkshashSchemaVersionagainstHASH_SCHEMA_VERSIONbefore anything else, so a layout change fails with an explicit "bump and regenerate" message rather than 100+ opaque mismatches.vitest.config.tsexcludestests/determinism/**from the defaultpnpm test;pnpm test:determinismruns it explicitly. New CI job Determinism, matrix[ubuntu-latest, windows-latest],continue-on-error: true, not in anyneeds:chain — never blocks a merge.tests/stream-usage.test.ts(regular, fast suite): a static scan assertingsystems/andworld/never call.get('genetics'|'reproduction'|'learning'|'events'). This is deliberately not a hash-based check — a system silently pulling from a reserved stream would still be perfectly reproducible (same seed, same result), so the determinism tests alone can't catch that "isolation" regression; this closes that gap the roadmap's testing checklist asks for.packages/simulation's tsconfig (tsconfig.jsonnow src-only; newtsconfig.test.jsonadds@types/nodefortests/) so the new tests can usefs/path/url, without weakeningsrc/'s no-Node/no-browser guarantee — verified with a negative check (process.envinsrc/still fails typecheck).Simulation impact
Determinism
Testing
Fresh clone of the pushed branch:
install --frozen-lockfile,format:check,check(191 fast tests, unchanged from before this PR),build, andpnpm test:determinism(15 tests, ~53s) all pass.hashSchemaVersionfails with the intended message; doubling the metabolism cost formula in a real system file fails 5 of 6 golden entries (the 6th has zero organisms, so it's correctly insensitive) — proof the suite is a real regression guard, not a tautology;process.envadded tosrc/index.tsstill failstsconfig.json's typecheck after the tsconfig split.vitest.determinism.config.tssetstestTimeout: 60_000.Review notes
This closes Phase 2: a seeded world runs 100,000 ticks headlessly, same seed -> same hash, different seed -> different hash, no NaN, food/starvation/death events work, no browser/Node dependency in the engine, and math/time/randomness/events all have unit tests.
🤖 Generated with Claude Code