fix(runner): report skipped concurrent instances as skips, not passes - #77
Merged
Drownek merged 5 commits intoSep 13, 2026
Merged
Conversation
A serial block instance that stopped early reported its remaining tests as `passed: true, durationMs: 0`, and `aggregateInstances` copied that into the aggregate's `instances` array verbatim. A skip was then indistinguishable from an instant pass for every consumer of that array. Skipped instances also lost their bot's name, which the reporter renders as "?". Adds `skipped`/`skipReason` to `TestInstanceResult` and sets `botUsername` on skip rows. Rendering is unchanged here; see the follow-up.
The per-test instance ratio counted skipped instances as passes, so a block that stopped early on two of ten bots still printed "[8/10]" with two of those eight never having run the test. The detail list printed the same instances as "OK (0ms)" under a "?" label, and their zeros pulled the min/avg duration down. The ratio is now over the instances that ran, with the skip count alongside it; skipped instances render as SKIP with their stop reason and no duration. The JSON report carries `skipped`/`skipReason` per instance.
A test row only counts as skipped when every one of its concurrent instances skipped it, so a run where several bots stopped their block early still printed "Skipped: 0" with nothing anywhere saying otherwise. The console summary now notes how many concurrent instances were skipped next to the test count, and the JSON report carries the same number as `summary.skippedInstances`.
"Expected message matching X not received" reads the same whether the server answered something else or said nothing at all, which is the whole question when a test fails on some concurrent instances and passes on others. The failure now appends the last 10 lines of the buffer it searched, or says outright that nothing was received.
Covers the new `skipped`/`skipReason` fields on each concurrent instance, `summary.skippedInstances`, and why a skipped instance's `passed: true` must not be read as a pass.
Drownek
approved these changes
Sep 13, 2026
Drownek
left a comment
Owner
There was a problem hiding this comment.
Great catch, clean implementation.
What I checked:
- Instance reporting: propagating
skippedandskipReasonthroughaggregateInstancesfixes the false-positive pass counts.[7/10]now shows as[5/8, 2 skipped], which is correct. - Duration stats:
ranInstances()keeps(0ms)skipped placeholders out of themin/avgcalculations, so they no longer drag the numbers down. - Bot attribution: passing
botUsernameon skipped serial positions gets rid of the anonymous?in failure traces. - Assertion diagnostics: quoting the trailing 10 lines of the message buffer in
toHaveReceivedMessage(or saying outright that nothing was received) makes it way easier to tell whether a command got throttled or the plugin just replied with something unexpected. Good call. - JSON reporter schema stays backwards compatible —
summary.skippedInstancesis purely additive.
Ran it locally across a few concurrent runs, no issues.
LGTM
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Under
concurrency: N, adescribe.serialblock that stops early on some of itsinstances reported the positions those instances never reached as passes. Below
is the shape of a real 10-bot run, with the plugin's names replaced:
Instances 4 and 6 had stopped at an earlier test in the block and never ran this
one. They still counted toward
7/10, printed asOK (0ms), dragged themindown to zero, and had no name to show. The summary said
Skipped: 0for thewhole run.
What changed
TestInstanceResultgainsskippedandskipReason, andaggregateInstancescopies them off the per-instance
TestResultinstead of dropping them. Skip rowsfrom a serial block now carry the bot's username too — that's where the
?camefrom.
The reporter reads the flag:
The ratio is over the instances that ran. Duration statistics skip the zeros. A
skipped instance shows its stop reason and no duration, because
(0ms)next to askip is the exact thing that read as an instant pass.
A test row only counts as skipped when every instance skipped it, so
Skipped:alone can't show this. The console summary andsummary.skippedInstancesin the JSON report now carry the instance count.Also here
toHaveReceivedMessagefailures quote the last 10 lines of the buffer theysearched, or say outright that nothing arrived. "Expected message matching
Already claimednot received" reads identically whether the server answeredsomething else or stayed silent, and on a live server that is the difference
between a throttled command and a plugin bug.
Verification
npm run typecheckandnpm run buildinrunner-package. Reporter outputchecked against a synthetic result reproducing the run above; JSON report
inspected for the new fields. No behavioural change to test execution — this
touches how results are recorded and printed, nothing about what runs.