Add scuzz diff: behavioral diff of a git revision against the working tree - #165
Merged
Merged
Conversation
Co-authored-by: sean.cheatham <sean.cheatham@gmail.com>
Co-authored-by: sean.cheatham <sean.cheatham@gmail.com>
…stRuntime. Co-authored-by: sean.cheatham <sean.cheatham@gmail.com>
Co-authored-by: sean.cheatham <sean.cheatham@gmail.com>
Co-authored-by: sean.cheatham <sean.cheatham@gmail.com>
Co-authored-by: sean.cheatham <sean.cheatham@gmail.com>
…odegen bug. Co-authored-by: sean.cheatham <sean.cheatham@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
scuzz diff [REV] [PATH]compares the behavior of a git revision (defaultHEAD) with the working tree under TestRuntime.What it does
examples/compiler/src/Delta.scuzz, pure Scuzz): parses timeline v=3 and reports the first diverging state, the number of differing states, length changes, and keyed changes per section (signalsby name,a11y/effects/filesas multiset lines, token sections by key). It renders the result as text or JSON. Oracles inexamples/tyck/delta.scuzz_verifycheck that a self-diff is empty, that swapping the sides swaps before and after, and that a prefix divergence survives appends.examples/compiler/src/Diff.scuzz):REVis checked out in a detachedgit worktreeunderbuild/delta/rev-<sha>. The same compiler builds both sides intobuild/delta/aandbuild/delta/b, and each side is cached by fingerprint. The command fails with a clear message when git is missing, the revision is unknown, or the path is not inside a work tree.corpus/directories, deduplicated by content hash. Each workload is classified assame,diverged,fixed,regressed, orunrunnable.unrunnablecovers a missing control or driver, a skipped script event, or schedule drift.Property.sometimesnames reached on only one side; shifts in classify counts.--iterations N,--seed): generates inputs over the alphabet both sides share and targets a timeline or pass/fail difference. It shrinks each hit with the existing shrinker, which now takes a predicate (Drive.fuzzShrinkBy), and then shrinks the fault seed andpct_k. Witnesses go tobuild/delta/witness-<hash>.toml. The command never writes tocorpus/.build/delta/report.json(v:1, kind:"diff"). Human output is ranked with regressions first.--message-format=jsonprints the report. The exit code is 1 only when there is a regression.Cmd.Diff,helpDiff, and a parse fixture (cliDiffArgs). A lone package-directory argument is read asPATH.scripts/ci-delta.shis a newdeltaslice, wired intoci.sh(pr,linux-headless) and thefuzzmatrix row inci.yml. It copiesexamples/counterandexamples/sharedinto a temp git repo and checks these cases:a11y, exit 0+1handler: regressed, andafterPlusShowsOnechanges from pass to fail, exit 1--iterations 16: witnesses failscuzz fuzz --replayon the working tree and pass on theREVworktree, andcorpus/stays unchangedverifyandcommandstopics (anddiffadded to the manual command verbs),philosophy.mdverification posture,vision.md, andgaps.mdLater.Found along the way
gaps.md, not fixed): when a lambda is passed directly to a user-defined higher-order def, the compiled lambda body reads the wrong value from a record field of its own parameter. The evaluator is correct. Repro:go(r => IO.pure(Str.len(r.script)), Run("p", "abc")). The first version of the shrinker refactor triggered this bug, andci-fuzz.shcaught it (the overflow-edge repro stopped shrinking). The shrinker now passes the run to a named def (fuzzShrinkProbe).cliFuzzoracle: it expected ashowstring withoutsession=false, so it failed on the base branch. The oracle now expectssession=false.Verification
macos-smokepass.linux-fuzzruns the fullci-fuzz.shand thendelta ok(31 s).scuzz checkandscuzz fmt --checkon compiler, cli, tyck, and manual: no errorsscuzz fuzz --iterations 0onexamples/tyck(delta oracles),examples/manual, and a copy ofexamples/cli: ok