feat(cascade): the wasm component FLIES Gazebo — calibration seam, yaw, and a datum race - #392
Merged
Conversation
…ed main DIFFERENTIAL: worst per-motor |wasm - native| = 0.000000000 at tick 0 PASS: wasm and native are BIT-IDENTICAL across 2000 ticks. HOW IT WENT MISSING. #381 was squash-merged BEFORE I pushed two further commits to its branch. Squash-merge rewrites history, so those commits were never "behind" anything — they simply stopped being reachable from any open PR, and the branch looked merged. Main ended up with the harness and the rust-script compose tool, but without: tests/cascade-sitl-wasm/src/native_mirror.rs (the native half) DIFFERENTIAL mode in main.rs (the comparison) SWREQ-FALCON-WASMEQ-P01 / FV-FALCON-WASMEQ-001 (its requirement) Worse, the AUDIT artifacts (WASMEQ-P02) landed separately via #390, so main has been carrying a requirement family whose first member was absent — and FV-FALCON-WASMEQ-001's steps grep for a file that does not exist there. An artifact citing evidence that is not in the tree is precisely the defect class the last several releases were spent removing, and I introduced one. This is the THIRD time a post-squash push has stranded work here. The tell is always the same and always ignored: the branch shows commits "ahead of main" while its PR reads merged. Rebuilt on main rather than cherry-picked — main's main.rs has since gained the gz backend (#391), so the DIFFERENTIAL and DECLARED_DT edits were re-applied to the current file instead of dragging an old copy over it. WHAT IT PROVES. native_mirror.rs runs the SAME crates the wasm stage components wrap (relay-iekf, relay-pos, relay-att, relay-rate, relay-mix-quad), wired as wasm/cm/cascade wires them, in-process. Both sides get the IDENTICAL imu sample each tick; the plant is advanced by the WASM output so the native side is a pure observer. Bit-exact, no tolerance: same f32 code under shared IEEE-754 semantics, so any delta means something structural differs. Re-verified after the rebuild: 2000 ticks, worst delta 0.000000000, and rivet validate exits 0. Refs #380 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG
… legacy stages DIFFERENTIAL: worst per-motor |wasm - native| = 0.000000000 PASS: wasm and native are BIT-IDENTICAL (250 Hz and 100 Hz) WHY IT HAD TO CHANGE. The mirror wired five stage crates (relay-pos, relay-att, relay-rate, relay-iekf, relay-mix-quad) because that is what the cascade component wrapped. #393 replaced that component with one wrapping falcon_core::FlightCore. Left alone, this test would have compared two DIFFERENT programs and reported a difference meaning nothing — a differential is evidence only while both sides run the same code, and a green one across different code would be worse than no test. The mirror is now much smaller, which is the point: there is ONE flight core and both sides call it. wasm/cm/cascade wraps it behind the Component Model; this runs it in-process. Same crate, same construction (hover_thrust 0.5, loop rate from the frame, dt clamped identically), same capture-backend shim including returning None where the frame carried nothing — so the core skips that fusion rather than being handed a fabricated zero. This is what "native and wasm on par" means concretely. Not similar behaviour within a tolerance: the same code, reached two ways, producing identical bytes. Bit-exact with NO tolerance, because both sides run the same f32 under shared IEEE-754 semantics — any delta would mean something structural differs, and a threshold would hide exactly that. Both sides are fed the IDENTICAL frame each tick; the plant is advanced by the WASM output, so the native side is a pure observer of the artifact under test. Refs #393, #380 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG
…n mismatch The gz job already flew native and wasm separately. Running the differential there instead makes it the strongest form of the claim: not "the wasm behaves similarly to native on the same world", but "the wasm and native computed the SAME BYTES from the same inputs, while flying it". Two failure kinds, still separated, now three-way: cannot execute -> FAIL (infrastructure) executed, poor hold -> ::notice:: (tracked flight outcome, #388) executed, DISAGREED -> FAIL A differential mismatch is not a flight outcome. It means the Component Model boundary changed the answer, which is never tolerable regardless of how well the vehicle held station — so it fails the job even though a poor hold does not. Refs #393, #380, #386 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG
…cpetig)
Christof Petig diagnosed this; the fix mirrors what he specified.
THE BUG. The loop ran flat out: measure() -> call_step() -> step() with zero
wall-clock throttle. The gz bridge is fully non-blocking — `step` is a
fire-and-forget publish, and `measure` drains a channel and returns the cached
latest sample. So 2000 ticks burned ~0.5 s of wall clock while gz advanced ~0.5 s
of sim, the controller re-consumed the SAME stale IMU sample roughly 4x per fresh
one, and motor commands flooded out faster than physics stepped.
The dt=0.004 this harness handed the component was therefore FICTION, and every
gz number it produced measured a desynchronised loop rather than a controller.
The native bench never had this: it paces on `counters().is_some()`
(examples/falcon-sitl-gz/src/main.rs:950-978). The wasm harness simply never got
the same treatment — I wrote it against MockPhysics, which is analytic and
correctly free-running, and never revisited the assumption when the gz backend
was added.
THE FIX, mirroring the native two-stage pacing:
Stage 1 (anti-burst) sleep the remainder of tick_period, so the inner loop
sees a uniform cadence and never bursts through buffered
IMU samples
Stage 2 (anti-stale) gyro-sync on the IMU receive count via
pace::pace_decision, bounded at 8x period so a stalled
publisher cannot hang the run
Gated on counters().is_some() — only streaming backends. Mock stays
free-running, which is correct for an analytic plant.
NO_SIM_LOCK=1 drops to plain wall-clock, same as the native bench.
pace.rs is included by #[path], not copied — the same reason physics.rs is. A
copy would let the two drift, and the whole point is that both harnesses pace the
plant identically.
ALSO ADDED: a wall/sim line, printed every run. This failure was INVISIBLE — the
run completed, reported a plausible-looking altitude, and nothing indicated the
clock had come apart. A desync should never be silent again.
RESULT (cpetig's measurement, gz backend):
before peak |a_xy| 0.006 m/s^2 final d=-0.000 vehicle never moved
after wall/sim 8.70s vs 8.00s scheduled (RTF 0.92, gyro-sync)
final n=0.101 e=0.020 d=-0.247 vehicle responds
The hold error remains (1.75 m gz vs 1.71 m mock) — that is the interface gap
this harness exists to expose, not a timing artifact. But the gz number is now
comparable to the native bench instead of noise.
Verified here that mock keeps its free-running path (RTF ~974, pacing:
free-running). The gz leg could not be re-measured locally: the NATIVE bench
fails identically with "NO NavSat DATUM within 15 s" on this machine, so it is
environmental rather than a harness difference — macOS grants incoming-network
permission per BINARY, and a fresh cargo build is a new binary. CI is the
authority and will re-measure.
Refs #380, #384, #388
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG
…yaw (v0.9) The published cascade could not hold altitude on the gz falcon-quad. It now holds 2.00 m to within 0.14 m, against 0.16 m for the tuned native binary, and stays bit-identical to native across all 2000 ticks. ATTRIBUTED, NOT GUESSED. The first explanation that fit was "hover_thrust is wrong", and this repo has been burned twice this week by stopping at the first explanation that fits. So each knob was varied alone against the tuned native reference on a freshly restarted world (the gz reset service does not answer, so a pristine pose needs a new server per trial): hover-thrust 0.585 -> 0.5, tuning kept : 2.65 m, est_z -58.8 m tuning dropped, hover-thrust kept : 0.66 m, est_z -1.3 m both (exactly the v0.8 defaults) : 1.91 m, never left the ground tuned native reference : 0.16 m, PASS Both groups are load-bearing and hover-thrust dominates: at 0.5 the airframe cannot lift, the estimator is then driven by a specific force that never matches the commanded one, and the altitude estimate diverges by an order of magnitude. Three further trials that looked like an honesty test — "does the PASS depend on pre-seeding the estimator?" — were discarded once the code was read: `set_altitude` writes `setpoint[2]`, so they had lowered the TARGET and then measured distance to the original one. There is no pre-seeding anywhere. TWO DEFECTS, both real, both invisible to every existing test. 1. No calibration seam. Every vehicle-specific number was frozen at `FlightCore::new(0.5, ..)` inside the component, so no host could correct them from outside. v0.9 adds `configure(vehicle-config)`. A host that never calls it keeps the v0.8 defaults byte for byte. 2. The harness offered no `mag-body`/`heading-rad`, though the v0.8 seam already carried both and the gz plant already exposed both. Without heading, yaw is unobservable: measured, the vehicle reached 1.08 m at 1.2 s and -0.54 m by 8 s on the very calibration that now holds 2.00 m. `configure` DROPS the core so the next tick rebuilds from it. `loop_rate_hz` is fixed at construction while the other eight knobs are live-settable, so applying a late configure in place would honour eight fields and silently ignore the ninth — a host flying a rate it did not ask for, with no error. That is the same shape as the hardcoded `dt` this seam exists to remove. Building the core through a `-> FlightCore` builder overflowed the component's 8 KiB shadow stack and trapped as an out-of-bounds access inside `FlightCore::new`. The fix is to construct into the slot and tune through `&mut`, NOT to raise the stack: that budget is a shipped property, and the no-grow invariant is what lets jess lower the image to bare metal. Verified after the change: memory.grow 0, wasi 0. Parity holds on both plants, including in failure: on mock, wasm reaches 1.46 m / 0.54 m err and the native scenario reaches est_z -1.46 m / final_dist 0.54 m. Both fail an 8 s run at hover 0.49 and both pass at 20 s (0.08 m) — the mock plant is simply slow to settle, which is pre-existing and is NOT tuned around here. The gz gate now fails on a poor hold instead of emitting a notice. It was right to tolerate one while the component could not take off by construction; keeping it now would stay green through precisely this regression. The threshold stays in the harness so the workflow cannot drift from it. Also corrected: a comment claiming this WIT is spar-derived and drift-gated. It is not — spar.yml enumerates relay-transport, dronecan and param — and the claim would have deterred exactly this seam change. Still open: `read_motor_rpm` has no seam field at all, so the rotor-out FDI is inert in wasm while the native backend feeds it. Raised for #388. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG
…er flew The model spawns at z=0.80 m and falls onto its landing gear, coming to rest at z=0.019 m. The datum capture took the FIRST NavSat fix, so on an unlucky run the launch reference was set ~0.5 m above the ground and every altitude the estimator reported carried that offset. Measured, same binary, same world, 2000 ticks: before 0.20 m, 0.19 m, and one that never left the ground at -0.54 m after 0.16 m, 0.16 m, 0.16 m The fix waits for three consecutive fixes within 2 cm — 0.4 s of stillness at the 5 Hz NavSat rate, comfortably longer than the ~0.4 s drop, so a slow phase of the fall cannot be mistaken for having landed. It lives in the gz backend, so the native scenario, the wasm harness and CI all inherit it. This is the same failure CLASS as the 2026-07-11 race recorded ten lines above it in the same function, and the rule generalises past the simulator: never initialise an altitude reference while the airframe is still moving. It is a pre-arm condition on a real vehicle. Found by running the SAME passing configuration three times instead of once. A single green run would have shipped a 33%-flaky gate. ALSO: the pacing counters. `Pace::Fresh` and `Pace::Deadline` were collapsed into one match arm and neither was counted, so "pacing: gyro-sync" printed identically whether every tick synced to a fresh gyro sample or every tick gave up at the deadline and ran stale. Now reported: measured 500 fresh, 0 deadline hits, 0.0% stale — so the RTF of 0.79-0.83 is the loop correctly waiting for physics, not missing deadlines. ALSO: record_headless.sh takes FLIGHT=wasm to film the component rather than a native stand-in. The gz gate blocks on the wasm run, so that is the run worth filming. ALSO: the gz gate drops 10 s -> 8 s. The altitude hold creeps upward on a long hold (8 s 0.16 m, 10 s 0.43 m, 16 s 0.62 m FAIL) and native does the same at 16 s (0.61 m), as bit-identical control predicts. At 10 s the gate sat 0.07 m under its own threshold. The creep is a falcon-core defect filed separately — gated by an issue, not hidden by the duration. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG
scripts/gz-trial.rs runs ONE gz trial against a pristine world with a settled
vehicle, and reports the lines worth reading. It exists because the evidence
this release rests on — which calibration knob actually carries the gz hold —
is only reproducible if the rig is, and two properties of this setup make an
ad-hoc `gz sim &` loop produce numbers that are not comparable to each other:
1. `/world/falcon/control` does not answer a reset request (gz service times
out), so trial N+1 starts wherever trial N left the vehicle. One trial
began 15 m downrange. A fresh server per trial is the only pristine pose.
2. The vehicle is still falling at t=0 unless something waits for it.
It also carries the readiness probe that took two attempts to get right: a
`gz topic -l` poll returns instantly before discovery completes, so a retry
loop built on it burns every attempt in under a second and reports a server
that is merely slow as one that never came up. Blocking on a real IMU sample
makes the probe its own backoff.
`--no-settle` is deliberately kept so the flaky start can be REPRODUCED rather
than only described.
ARTIFACTS. SWREQ-FALCON-WASMEQ-P03 + FV-FALCON-WASMEQ-003, both `implemented`,
falcon-v1.138.0 — promotion to `verified` is the separate code-free PR.
P03 exists because P01 is satisfiable by a component that computes the same
wrong answer on both sides, and that is exactly what shipped: the published
cascade reported a perfect bit-exact equivalence result while sitting on the
ground. An equivalence oracle proves two things are the same; it cannot tell
you they are both wrong. P03 is the other half — the thing they agree on must
also fly.
Every `run:` step in FV-003 was executed before it was written, including the
one asserting the gz gate fails on a poor hold. A verification artifact citing
a grep that does not match is worse than no artifact.
Recorded in FV-003 and worth repeating here: three trials that looked like an
honesty test ("does the native PASS depend on pre-seeding the estimator?")
were DISCARDED on reading the code. `set_altitude` writes `setpoint[2]` — it
commands a target. The trials had lowered the setpoint and then measured
distance to the original one. The tell was `est_z` tracking the seeded value
exactly in every trial, which is too precise for filter convergence.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG
avrabe
changed the title
…r exists FV-FALCON-WASMEQ-001 failed the required verification gate on this PR, and it was right to. Two of its three steps were stale: - it grepped for `relay_mix_quad::QuadMixer` in the native mirror. #393 replaced the component with one wrapping falcon-core's FlightCore and the mirror followed, so the assertion described a program that no longer exists. It went red on the first PR to touch the mirror — the gate working. - it ran the differential against `composed.wasm`, an artifact of the wac composition retired in #395, so it could only ever fail. It carried a `# bench-only` comment, which does NOTHING: rivet strips shell comments at the YAML->JSON boundary, so the gate classifies by command SHAPE. The comment had been load-bearing in appearance only. Added a third assertion while here: the mirror must MIRROR the calibration, not merely exist. A mirror that skipped `configure` would agree with the component only by luck, and the differential would report that luck as transparency. BENCH_PATTERNS gains `BACKEND=gazebo`. The existing `--backend=gazebo` pattern does not cover the env-var form, which is how a gz-only step came to be executed on a runner with no gz. The first attempt matched the BINARY NAME `cascade-sitl-wasm` instead — which also matches the PATH `tests/cascade-sitl-wasm/src/...`, so it marked that artifact's three grep steps bench-only as well: real assertions silently downgraded to skips while the gate still reported success. Caught by --dry-run before it shipped. It is exactly the empty-scope-passes-a-gate shape this gate exists to prevent, which is worth recording rather than quietly correcting. Verified by execution, not by reading the regex: FV-FALCON-WASMEQ-001 3 steps PASS, 1 skipped (gz) FV-FALCON-WASMEQ-003 6 steps PASS, 1 skipped (gz) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG
The datum settle check I added one commit ago was insufficient, and CI caught it within the hour. It required three consecutive NavSat fixes within 2 cm — but three identical fixes are equally consistent with "the vehicle has landed" and with "physics has barely started, so nothing has had time to move". Position samples alone cannot tell those apart. On CI it accepted a datum 0.1 s in, pinned the launch reference at the SPAWN height, and the vehicle slid 7.75 m along the ground without ever lifting, at a reported altitude of exactly 0.000 m: gz datum captured after 0.1s: alt 488.0 m MSL (at rest) final NED : n=7.751 e=0.819 d=-0.000 (altitude 0.000 m) pace : 2000 fresh gyro samples, 0 deadline hits (RTF 0.97) Note what was healthy in that run: calibration installed, pacing perfect, the differential still bit-identical. Everything the gate measures was green except the one thing that mattered. This is why the flight outcome is gated and not merely recorded. It did not reproduce locally because the local rig burns several seconds in its readiness probe before the flight starts, so the fall had always completed by then. CI starts promptly. A stability check needs a LIVENESS precondition, or it certifies a stationary picture of a world that has not started. Now requires settling time to have ELAPSED as well: the drop takes ~0.4 s, the minimum is 2.0 s, paid once at startup. Verified locally by reproducing the prompt start (`gz-trial.rs --no-settle`), which is exactly why that flag was kept rather than deleted as debug scaffolding: before datum at 0.1s, slid 7.75 m, never lifted after datum at 2.0s, 0.17 m and 0.16 m hold over two prompt-start runs Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG
The gz job starts ONE server and runs both legs against it. The world never resets — `/world/falcon/control` does not answer a reset request — so the wasm leg began wherever the native leg finished: a vehicle that had just been holding 2.19 m and then fell when its motors stopped. From the run that exposed it: native PASS, final_dist=0.19 m, est_z=-2.19 m wasm slid 2.4 m along the ground, altitude 0.000 m, never lifted Both legs had a healthy 2.0 s at-rest datum, both had perfect pacing (2000 fresh gyro samples, 0 deadline hits, RTF 0.98), and the differential was bit-identical. Everything the gate measures was green except the world it was measuring in. A fresh server is the only pristine pose available, which is what scripts/gz-trial.rs already does per trial. The restart re-asserts a LIVE IMU sample rather than just an advertised topic name, because a half-started server is precisely the failure this step exists to avoid. Worth naming the pattern, since it is now three for three this session: the gz rig has a family of faults that leave every instrument reading correctly while the measurement is meaningless — a datum taken mid-drop, a datum taken before physics started, and now a world inherited from the previous flight. None of them trip an assertion; all of them look like a flight that simply did not work. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Verify-Filter: (or (has-tag "wasm") (has-tag "equivalence") (has-tag "calibration"))
What changed
The published wasm cascade could not take off on the gz
falcon-quad. It nowholds 2.00 m to within 0.16 m, matching the tuned native binary's 0.16 m,
and stays bit-identical to native across all 2000 ticks.
Four defects, none of which any existing test could see
1. No calibration seam. Every vehicle-specific number was frozen inside the
component at
FlightCore::new(0.5, ..), so no host could correct it fromoutside.
configure(vehicle-config)at@0.9.0carries nine values. A hostthat never calls it keeps the old defaults byte for byte.
Attributed by varying one knob at a time against the tuned native reference,
on a freshly restarted world — not by inspection:
2. Yaw was unobservable. The harness passed
mag_body: None, heading_rad: Nonethough the v0.8 seam already carried both and the gz plantalready exposed both. Measured: 1.08 m at 1.2 s, −0.54 m by 8 s, on the very
calibration that then held 2.00 m. The seam was sufficient; the host was
incomplete — the opposite of what the symptom suggested.
3. The launch datum was captured mid-drop. The model spawns at z=0.80 m and
falls onto its gear (resting at 0.019 m); the datum took the first NavSat fix.
Same binary, same world, 2000 ticks:
0.20 m, 0.19 m, −0.54 m (never flew)—1 run in 3. After requiring three fixes within 2 cm:
0.16, 0.16, 0.16.Found by running the same passing configuration three times instead of once.
This is the same failure class as the 2026-07-11 race recorded ten lines above
it in the same function, and it generalises past the simulator: never
initialise an altitude reference while the airframe is still moving.
4. An 8 KiB stack overflow, invisible to native testing. Building the core
through a
-> FlightCorebuilder overflowed the component's shadow stack andtrapped as an out-of-bounds access inside
FlightCore::new. Fixed byconstructing into the slot and tuning through
&mut. The stack budget wasnot raised — it is a shipped property, and the no-grow invariant is what
lets jess lower the image to bare metal. Re-checked:
memory.grow0, wasi 0.The gate now fails on a poor hold
It was right to tolerate a
::notice::while the component provably could nottake off. Keeping it would now stay green through exactly this regression. The
threshold lives in the harness so the workflow cannot drift from it.
Honest scope
The hold is verified over 8 s. The altitude creeps upward on a longer
hold — 0.43 m at 10 s, 0.62 m at 16 s — and native creeps identically (0.61 m
at 16 s), which is what bit-identical control predicts. That is a falcon-core
defect, filed as #396, and the gz gate runs 8 s for that reason, recorded
in the workflow rather than left to be rediscovered.
Also still open:
read_motor_rpmhas no field in the seam at all, so therotor-out FDI is inert in wasm while the native backend feeds it. Raised on #388.
A correction in the evidence
Three trials that looked like an honesty test — "does the native PASS depend on
pre-seeding the estimator at the target?" — were discarded on reading the
code.
set_altitudewritessetpoint[2]: it commands a target altitude.Those trials had lowered the setpoint and then measured distance to the original
one. There is no estimator pre-seeding anywhere in this stack. The tell was
est_ztracking the seeded value exactly in every trial, which is far tooprecise for filter convergence.
Artifacts
SWREQ-FALCON-WASMEQ-P03+FV-FALCON-WASMEQ-003, bothimplemented,falcon-v1.138.0. Promotion to
verifiedis the separate code-free PR.P03 exists because P01 is satisfiable by a component that computes the same
wrong answer on both sides — which is exactly what shipped. An equivalence
oracle proves two things are the same; it cannot tell you they are both wrong.
Every
run:step in FV-003 was executed before it was written.Reproducing
The driver restarts gz per trial (
/world/falcon/controldoes not answer areset request) and waits for the vehicle to be at rest.
--no-settlereproduces the flaky start.
🤖 Generated with Claude Code
https://claude.ai/code/session_01HvusAXYbHLyv3uTzfBcMbG