Skip to content

Three contended_up tests stage their race on a 30ms sleep and flake under load #633

Description

@blooop

Three tests built on contended_up (rust/devlaunch-core/src/flows/launch.rs) go
red under load, about 5 times in 84 full --lib runs at --test-threads=24 to
48:

  • a_contended_up_still_skips_for_a_sibling_whose_create_finished
  • a_sibling_that_won_the_race_still_leaves_this_launch_a_copy
  • a_launch_that_waited_for_its_prewarm_is_partial

They fail as Ok(Started) where Ok(SkippedSiblingWon) was expected.

Why

contended_up stages the race on a timer. The parent takes the launch lock,
spawns the launch thread, sleeps 30ms, then drops the lock. The 30ms is a guess
about how long the child needs to reach its flock. Under load the child has not
got there yet, so it takes an uncontended lock, waited() is false, and the skip
the test exists to observe never applies. Nothing is wrong with the code under
test; the test's own staging is what slipped.

The fix worth making

Release the parent's lock on the child's WaitingForSiblingLaunch notice rather
than on a timer. The notice is the event the sleep is approximating, and it is
already raised on the path these tests drive, so the race can be staged
deterministically instead of probabilistically.

Provenance

Pre-existing: introduced with the contention tests themselves (32fb1ae, 7fb09b8,
3ceddea), all well before the branch this was found from. Found while chasing a
different, unrelated report during a self-review.

Not the same flake as
#526, which is
the_json_listing_migrates_the_cache_and_the_table_does_not.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions