Skip to content

Flaky wall-clock scaling tests can fail CI and the release gate #7

Description

@rahlk

Problem

TestTemplateDenseSingleLineParsingScalesNearLinearly (and its sibling TestTemplateParsingScalesNearLinearly) guard against an O(n²) regression by comparing wall-clock time of a 4× larger input against small*9 + 50ms (small*8 + 25ms). On a shared GitHub runner the small measurement lands near 12 ms, so the bound is ~160 ms and ordinary scheduler/GC noise on the large run trips it. Seen on PR #6 (ci job "full suite with a graph", run 34002119560): 8k dense same-line actions took 166ms after 2k took 12ms; want <= 158ms. PR #6 changes no Go code.

Because release.yml runs make test as its hard gate and deletes the tag on failure, this flake can abort a release.

Scope boundary

In: make both scaling tests robust to runner noise without weakening the quadratic-regression guard: larger inputs so the small measurement is well above the noise floor (target ≥ 50 ms on a runner), best-of-N kept, ratio bound kept, absolute ceiling kept.
Out: any change to the parser or to what the tests assert about spans/counts.

Goals

  • go test ./internal/dialects/helm -run ScalesNearLinearly -count=30 passes locally and the tests still fail on an injected quadratic regression (temporarily, to prove the guard).
  • Total added test time under one second.

Caveats and known risks

  • A too-large input hides a quadratic regression behind the absolute ceiling; keep the ratio check primary.

Definition of done

  • Both tests pass 30× locally and on CI; the regression guard is demonstrated in the PR description.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions