Skip to content

fix(test): wait for expired test teardown through its future - #172

Merged
wolfy-j merged 1 commit into
masterfrom
fix/test-runner-deadline-barrier
Oct 12, 2026
Merged

wolfy-j merged 1 commit into
masterfrom
fix/test-runner-deadline-barrier

Conversation

@wolfy-j

@wolfy-j wolfy-j commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Defect

The runner imposes an implicit 30-second deadline on test entries. On expiry it cancels the function future, waits only 400ms for settlement, and the parent waits only 500ms before advancing. Those waits cannot establish isolation: cancellation could close the future while the function was still tearing down.

Change

  • Use only the existing cancel() and response():receive() API. On expiry the wrapper requests cancellation, receives completion on the same future, and then acknowledges the parent. The parent waits for that acknowledgement before starting another entry or suite. There is no teardown timeout or capability probe.
  • Remove the implicit entry deadline. Existing meta.timeout remains an explicit entry budget.
  • Use meta.suite_timeout as one elapsed-time budget for all entries in a suite. Reject conflicting budgets and fail remaining entries without starting them once the suite expires, including when expiry happens during an earlier entry's teardown.
  • Preserve entry attribution and verdict checks; a real result racing with expiry also proves completion.

Requires the semantic fix in wippyai/runtime#961 to land first. It keeps cancellation non-blocking and publishes the actual future outcome only after execution-frame teardown. Older runtimes expose the same methods but do not provide this completion guarantee.

Verification

  • GOTOOLCHAIN=go1.27.1 GOMAXPROCS=4 make test-runner-go: race-enabled tests against the actual Lua runner sources and Go vet pass.
  • Deterministic failing-first coverage proves that an expired entry cannot acknowledge or start its successor while teardown is blocked. The regression fails on framework master; explicit budget, conflicting-budget, and expiry-during-teardown tests also pass with this change.
  • Targeted runner lint with the revised local runtime: 13 entries, no issues.
  • Both real timeout/isolation scenarios pass.
  • Runner verdict tests pass (17 cases), as do the result/protocol regression harness and the hook-abort harness.

Based on master, the framework repository's default branch.

@wolfy-j
wolfy-j requested a review from skhaz October 11, 2026 15:12
@wolfy-j

wolfy-j commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor Author

Bee proof committed locally at c2ab3d38 on proof/runtime-prs (no push).

native/proof:TestBeeRunnerDeadlineIsolation: The actual installed runner selects a controlled case deadline; the next listen/preparer fixture sees a real runtime future’s frame lease. Main advances uncanceled; #172/#961 join a cleanup barrier first. Patched wippy/test is vendored locally in the probe workspace. No sleeps.

20 focused runs: main 2a8f0b412ec3 0 passed / 20 failed; PR framework 9395420c + runtime 1ea38a097343 20 passed / 0 failed. Patched race checks and Go vet pass.

Completed full make test SHARDS=8 (384 entries): main 2819 passed / 6 failed, 412.0s; initial combined #958/#959/#960/#961 + framework #172 2827 passed / 0 failed, 362.5s (#961 at 210f6d0a3). The refresh at #961 1ea38a097343 is incomplete: 7/8 shards finished, Docker cleanup_test and Gov application_lifecycle_test hit deadlines, and unbounded Node upgrade_test stalled. Two earlier refresh attempts failed the unchanged 10s strict-boot budget. No tests or deadlines were weakened.

@wolfy-j
wolfy-j force-pushed the fix/test-runner-deadline-barrier branch from 9395420 to 8c013b9 Compare October 11, 2026 16:28
@wolfy-j

wolfy-j commented Oct 11, 2026

Copy link
Copy Markdown
Contributor Author

Revised: no new API; cancel() now completes the future only after teardown.

@wolfy-j wolfy-j changed the title fix(test): make deadlines explicit and join expired test teardown fix(test): wait for expired test teardown through its future Oct 11, 2026
@wolfy-j
wolfy-j merged commit 4486344 into master Oct 12, 2026
24 checks passed
@wolfy-j
wolfy-j deleted the fix/test-runner-deadline-barrier branch October 12, 2026 00:53
@wippy-releases wippy-releases Bot mentioned this pull request Oct 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants