Repository navigation
CTRL+C does not complete the test run when there is long running test #5345
Description
Activity
Not sure if this is related, but when I run all tests locally (xUnit v3, 1000+ tests, multi-targeted, many projects) and press Ctrl+C, nothing terminates. I don't know if it helps, but I have to press Ctrl+C like ten times in a row to get back to the command prompt.
This only happens with MTP. VSTest does not have this problem. I can imagine that cancelling a CI job faces similar problems (we auto-cancel existing jobs for a PR/branch when a new commit is pushed).
From my point of view, interpreting Ctrl+C as a cancellation request is not intuitive. It is supposed to work as a hard break, similar to pressing Stop in a debugger: kill whatever was running and exit immediately.
Yes, this seems like the same problem. In the description I tried to remedy this with 2 different approaches. But none of them were accepted in the code base.
Problem of why it hangs is described here: #5288 (comment)
Something somewhere needs to break the cooperative cancellation chain (otherwise we will wait forever for code that is not cooperatively cancelling), and based on the feedback in the PRs this is better done in the test framework, than in MTP. I don't disagree, it is better for the framework to choose the behavior than us forcing it. So this probably needs to flow better into xunit, and there the long running task that holds up the cancellation needs to be abandoned to the background and control returned back to the platform.
I'm not sure I fully understand, but waiting for a cancellation token is only reliable when everything fully supports cancellation. That is often not the case.
We pass
TestContext.Current.CancellationTokento all async calls in all tests. But we also have some code-under-test that performs sync-over-async without a token. The reason is thatIConfigurationProvider.Loadhas no async equivalent and doesn't accept a token. So if our configuration provider performs a network call and DNS is slow, it can take up to five seconds to break out (we often experience this on macOS). Or 30 seconds before the HTTP request times out.From my point of view, a Ctrl+C should just call something like
Environment.FailFast, instead of trying a graceful shutdown.Yeah, we are saying the same thing. 👍
My two PRs above implement something similar to stopping ungracefully. But a better place to put that code is in the framework and it's handling of cancellation code that does not cooperate. So the platform does not force a (theoretical) framework that is cooperative cancellation only, to abort threads and exit forcefully.
I am not against the Ctrl+C to do a fail fast on MTP, my main concern was and is only that there is always someone that wants to cancel fast but still have the trx, coverage or whatever report.
My point is to say we either cancel and assume everything cooperates or we terminate but we don't do some kind of graceful termination where we try to kill some stuff after some time so we can try to continue (we saw the number of issues related to this design on VSTest).
I think Jakub Jareš (@nohwnd) has/had one draft PR with first Ctrl +C triggering cancellation and second doing strong abort and this design is fine by me.
Okay I might have been misremembering. The thing we agreed on is actually described in Additional context in the issue above. Let's do that first step then :)
As I think I detailed in other places:
In my ideal world the cancellation would have multiple stages, it would notify the framework to start tearing down.
Framework could ask for additional time to tear down (e.g. when it wants to do additional cleanup via async calls, like playwright, by sending "cancellation ack, ETA 10s" ). This would be communicated to the user, e.g. Playwright is cancelling... After 10s we would see if Playwright asked for more time or not.
I don't think that is what currently happens in the platform, when cancellation happens, the test starts tearing down, and also the platform starts tearing down, giving no grace period to either the framework or the extensions to clean up.
Details of this are blurry for me though, and it is realized via cancellation tokens which as not easy to track where they propagate for this kind of flow.
Perhaps have two key bindings then: one for hard-stop and one for request-abort?
In the ReSharper Unit Test Sessions view in VS, there are actually two buttons: one to cancel the testrun, which doesn't always respond immediately. And there's a hard kill button, similar to the debugger-stop button. Usually, the first button is sufficient. But sometimes it never completes, so we need the second one. How it works internally, I don't know. If you're not already in contact with the R# team and want details on how that works, I can reach out to them.
We discussed that as well, but people usually understand ctrl+c as abort, and aborting right away is not great experience here, because cleanups won't run.
We have similar gestures in vstest as well, cancel and abort (but VS only aborts, because cancellation takes time and makes it look stuck - to me this is not great experience either, and is often source of complaints and confusion)
Posting this as a follow-up to my earlier comment and to give #8580 some design context.
RFC: Phased graceful shutdown for Microsoft.Testing.Platform
Tracks: #5345 · Related prior attempts: #5267, #5288 · Status: Draft
Summary
Replace MTP's single-shot
CancellationTokenmodel with a two-phase
shutdown (Draining → Aborting) bounded by a configurable grace ceiling, and
align the Ctrl+C UX with the convention used bydocker compose,kubectl,
npm,vitest, anddotnet watch: first press = cooperative cancel, second
press = abort, third press = terminate.This RFC describes Design A only. A follow-up RFC will cover an opt-in
IShutdownParticipantinterface for extension-side ETA negotiation
(Design B). An IPC/server-mode variant (Design C) is deferred.Motivation
Today MTP has exactly one cancellation signal: a
CancellationTokenSource
flipped by Ctrl+C
(CTRLPlusCCancellationTokenSource).
Every participant — framework, reporters, dump extensions, retry,
crash/hang dump — observes the same token at the same time. This causes
three recurring failure modes that show up across MSTest, xUnit v3, and
several CI scenarios:- Shutdown hangs forever. When a framework's cancellation is only
partly cooperative (e.g. xUnit v3 sync-over-async in
IConfigurationProvider.Load, blocking DNS calls), Ctrl+C does
nothing and the user has to press it 10× to recover. Reported on
CTRL+C does not complete the test run when there is long running test #5345 by external users. - Lost reports. Conversely, when shutdown does propagate, every
participant tears down simultaneously — there is no window to flush
TRX, coverage, AzureDevOps artifacts, or crash dumps. Users
experience either-or behaviour and cannot choose. - No observability. The user sees the cursor sit. They cannot tell
whether the platform is making progress, which extension is delaying
shutdown, or how long to wait before force-killing.
The previous attempts at "make it harder/faster" (#5267, #5288)
correctly identified the trade-off but did not give the test framework
or extensions a contract to participate in. This RFC proposes that
contract.Goals
- Ctrl+C reliably exits within a bounded time, even when frameworks
don't perfectly cooperate. - Well-behaved reporters get a chance to flush their output.
- The user sees what's happening and knows how to escape.
- CI builds get a deterministic shutdown ceiling they can configure.
- No breaking change to existing extensions — the new contract is
additive.
Non-goals
- Per-extension ETA negotiation (Design B follow-up).
- Cross-process / IPC shutdown messaging (Design C follow-up).
- Replacing
CancellationTokensemantics with something exotic — we
layer phases on top of standard tokens.
Design
Lifecycle phases
RUNNING ──Ctrl+C/cancel──▶ DRAINING ──grace elapsed──▶ ABORTING ──abort elapsed──▶ EXITED ▲ ▲ ▲ 2nd Ctrl+C ────────────────────┘ │ 3rd Ctrl+C ──────┘ (Environment.FailFast)Phase Meaning Triggered by Running Normal test execution. — Draining Stop accepting new work; finish in-flight; flush reporters. 1st Ctrl+C, --shutdown-grace-periodreachedAborting Stop everything cooperative; reporters should bail out quickly. 2nd Ctrl+C, grace period elapsed Exited Host shut down. natural exit, or 3rd Ctrl+C / abort elapsed Two tokens
The existing
ITestApplicationCancellationTokenSourcegains two named
tokens. The existingCancellationTokenproperty keeps its current
meaning (aliased toDrainingTokenfor source compatibility — no
existing consumer changes behaviour).public interface ITestApplicationCancellationTokenSource { /// <summary> /// Existing API. Equivalent to <see cref="DrainingToken"/>. /// Signalled when the platform enters the Draining phase. /// </summary> CancellationToken CancellationToken { get; } /// <summary> /// Signalled when entering Draining. Honour it by stopping the /// dispatch of new work and flushing in-flight state. You may take /// up to <c>--shutdown-grace-period</c> to finish. /// </summary> CancellationToken DrainingToken { get; } /// <summary> /// Signalled when entering Aborting. Honour it by bailing out of /// long-running work as fast as you can. You may take up to /// <c>--shutdown-abort-timeout</c> to finish; after that the host /// will terminate. /// </summary> CancellationToken AbortingToken { get; } void Cancel(); void CancelAfter(TimeSpan timeout); }
Internally,
DrainingTokenis the source of truth for "user wanted to
stop."AbortingTokenis a linked token derived fromDrainingToken- a delay (
shutdownGracePeriod) + a 2nd-Ctrl+C signal — whichever
trips first.
CLI options
Option Default Meaning --shutdown-grace-period30sMax time spent in Draining before auto-escalating to Aborting. Matches HostOptions.ShutdownTimeout.--shutdown-abort-timeout10sMax time spent in Aborting before the host force-terminates. Both accept a
TimeSpan(e.g.30s,2m,00:01:30). Both have
config-file equivalents under a newplatformOptions:shutdownsection.Ctrl+C UX
^C Cancelling test session… (press Ctrl+C again to force quit) ^C Aborting test session… (press Ctrl+C again to terminate immediately) ^C Terminated.The status line for Draining/Aborting is written by the existing
TerminalOutputDevice; in non-interactive output modes (CI logs,
JSON-RPC) it becomes a structured log event so consumers can render it
their way.Behavioural rules
- Entering Draining is idempotent. Multiple cancel sources
(Ctrl+C,--treenode-filterearly exit, framework error, server
request) all converge. - Entering Aborting is idempotent and irreversible. Once aborting,
you cannot return to draining. - The grace timer starts when Draining is entered and is not
restartable in Design A. (Restartable / per-extension extension
= Design B.) Environment.FailFasthappens only on the 3rd Ctrl+C or if the
abort timer also elapses. The platform itself never silently
FailFasts.- Extensions that observe
AbortingTokenand exit promptly cause no
FailFast. - Existing code that observes the old
CancellationTokenis
automatically treated as observingDrainingToken— no source
change required.
Host integration
CommonTestHostowns the phase state machine.TestHostBuilder.CommonServiceswires the new tokens into the
service container alongside the existing source.TestSessionLifetimeHandlersContainer.OnTestSessionFinishingAsyncis
invoked withDrainingToken(so reporters can still flush) and a
parallelAbortingTokenis exposed viaITestSessionContextfor the
rare extension that wants to differentiate.TestHostOchestratorHost/TestHostControllersTestHostpropagate
both tokens to controlled hosts via env vars
(MTP_SHUTDOWN_GRACE_PERIOD_MS,MTP_SHUTDOWN_ABORT_TIMEOUT_MS) so
child processes share the same budget.
Compatibility
- Source-compatible.
ITestApplicationCancellationTokenSource
gains two members. We will mark the new members as
[EditorBrowsable(Never)]for one release to keep IntelliSense
clean during migration. - Binary-compatible. Interface gains default-implementation
members; existing implementers (mocks, test infrastructure) are
unaffected. - Behavioural change: the platform now waits up to
--shutdown-grace-periodbefore bailing. Today it waits forever in
some cases and ~instantly in others. The new default (30s grace + 10s
abort) is a strict improvement for both the "waits forever" and
"kills reports" failure modes, but CI maintainers who rely on
"Ctrl+C = immediate exit" will need to set
--shutdown-grace-period 0s.
Alternatives considered
- Single-token + escalation flag. Reuse the current token; add a
bool IsAbortingon the source. Rejected because consumers cannot
subscribe to the transition with standardCancellationToken
primitives (Register,WaitHandle,WhenCanceled). - Three tokens (Cancelling / Aborting / Terminating). Considered,
but Terminating cannot be honoured — at that point the host is
FailFasting andRegistercallbacks won't run. Two tokens is the
right shape. - Negotiated ETA in this RFC. Defers cleaner UX, but adds API
surface that we'd have to live with forever. Better to ship the
floor (Design A) and prove the need for Design B with concrete
extension demand (Playwright bridge, large-trace coverage).
Prior art
System Notify Negotiation Hard kill Kubernetes preStop → SIGTERM terminationGracePeriodSecondsSIGKILL systemd SIGTERM sd_notify(EXTEND_TIMEOUT_USEC)SIGKILL AWS ECS SIGTERM stopTimeout30–120sSIGKILL AWS Lambda Extensions SHUTDOWNevent +deadlineMsNone (fixed 2s) runtime macOS AppKit applicationShouldTerminate:NSTerminateLater+ replyOS kill Windows SCM SERVICE_CONTROL_PRESHUTDOWNdwPreshutdownTimeoutprocess Erlang/OTP terminate/2shutdownstrategybrutal_kill .NET Generic Host IHostApplicationLifetime.StopAsyncHostOptions.ShutdownTimeout(default 30s)dispose Vitest SIGINT teardownTimeout(default 10s)process exit npm / docker compose / k9s 1st SIGINT cancel, 2nd kill — process exit ReSharper Test Sessions Stop button (cooperative) Kill button (separate) process kill Design A is the Kubernetes / .NET Generic Host shape: bounded
ceiling, no per-participant negotiation. Design B is the systemd
EXTEND_TIMEOUT_USECshape, layered on top.Rollout
- Land the token + phase machinery behind feature flag
--internal-experimental-phased-shutdown(off by default). - Wire the existing extensions (TRX, coverage, dump) to differentiate
Draining vs Aborting where it matters; verify behaviour with
acceptance tests. - Flip the default on for one preview release.
- Remove the flag.
Open questions
- Should the grace ceiling be inherited from
TestContext-style
configuration on the framework, or only platform-side? Today, MSTest
ignores platform timeouts. Probably worth a separate discussion. - For server-mode, does the client need a way to initiate the phase
transition (e.g. IDE "Stop" button)? Probably yes, and that's the
natural intro point for Design C. - Should
--shutdown-grace-period 0sshort-circuit straight to
Aborting, or is "0s means no grace" too surprising? Suggest the
former with a doc note.
- Shutdown hangs forever. When a framework's cancellation is only
Describe the bug
Running a project with test that takes long time or infinite time will not return control to the user in timely manner when they press Ctrl+C.
MTP does cooperative cancellation, and leaves the details to the test framework, and MSTest (and NUnit) do block the returning on main method, until the currently running test completes.
Steps To Reproduce
Have single test that sleeps for 10s.
Run tests as exe and cancel by pressing Ctrl+C.
Cancelling test session message is shown, but we still wait 10s for the test to finish.
Expected behavior
Run is cancelled / aborted.
Actual behavior
Described above.
Additional context
To solve this we agreed to implement a 2 stage approach. When user presses CTRL+C we will start cancelling the session. This is up to the test framework to handle. MTP will only cooperatively cancel the framework, and if the framework does not return control we cannot do anything.
The framework is free to abandon the test work, and continue to cleanup. To ensure fast cancellation. And cleanup.
We will also output message to Press Ctrl+C again to abort.
When user does that we will kill the process, to ensure timely return of control. But we won't guarantee any cleanup.