Three tests built on contended_up (rust/devlaunch-core/src/flows/launch.rs) go
red under load, about 5 times in 84 full --lib runs at --test-threads=24 to
48:
a_contended_up_still_skips_for_a_sibling_whose_create_finished
a_sibling_that_won_the_race_still_leaves_this_launch_a_copy
a_launch_that_waited_for_its_prewarm_is_partial
They fail as Ok(Started) where Ok(SkippedSiblingWon) was expected.
Why
contended_up stages the race on a timer. The parent takes the launch lock,
spawns the launch thread, sleeps 30ms, then drops the lock. The 30ms is a guess
about how long the child needs to reach its flock. Under load the child has not
got there yet, so it takes an uncontended lock, waited() is false, and the skip
the test exists to observe never applies. Nothing is wrong with the code under
test; the test's own staging is what slipped.
The fix worth making
Release the parent's lock on the child's WaitingForSiblingLaunch notice rather
than on a timer. The notice is the event the sleep is approximating, and it is
already raised on the path these tests drive, so the race can be staged
deterministically instead of probabilistically.
Provenance
Pre-existing: introduced with the contention tests themselves (32fb1ae, 7fb09b8,
3ceddea), all well before the branch this was found from. Found while chasing a
different, unrelated report during a self-review.
Not the same flake as
#526, which is
the_json_listing_migrates_the_cache_and_the_table_does_not.
Three tests built on
contended_up(rust/devlaunch-core/src/flows/launch.rs) gored under load, about 5 times in 84 full
--libruns at--test-threads=24to48:a_contended_up_still_skips_for_a_sibling_whose_create_finisheda_sibling_that_won_the_race_still_leaves_this_launch_a_copya_launch_that_waited_for_its_prewarm_is_partialThey fail as
Ok(Started)whereOk(SkippedSiblingWon)was expected.Why
contended_upstages the race on a timer. The parent takes the launch lock,spawns the launch thread, sleeps 30ms, then drops the lock. The 30ms is a guess
about how long the child needs to reach its
flock. Under load the child has notgot there yet, so it takes an uncontended lock,
waited()is false, and the skipthe test exists to observe never applies. Nothing is wrong with the code under
test; the test's own staging is what slipped.
The fix worth making
Release the parent's lock on the child's
WaitingForSiblingLaunchnotice ratherthan on a timer. The notice is the event the sleep is approximating, and it is
already raised on the path these tests drive, so the race can be staged
deterministically instead of probabilistically.
Provenance
Pre-existing: introduced with the contention tests themselves (32fb1ae, 7fb09b8,
3ceddea), all well before the branch this was found from. Found while chasing a
different, unrelated report during a self-review.
Not the same flake as
#526, which is
the_json_listing_migrates_the_cache_and_the_table_does_not.