FIX: Keep the backend responsive while starting scenario runs

[varunj-msft]

Description

POST /runs initializes everything eagerly, on purpose, so that configuration errors reach the caller instead of disappearing into a background task. The problem is where that work ran.

It ran directly on the event loop, and it is slow and almost entirely synchronous — loading the default datasets alone takes minutes. For that whole window the server answered nothing at all. Health probes timed out, and the CLI reported the server as unavailable even though it was alive and simply busy. That is the failure mode behind the current End to End Tests failures, where the client gives up before the server has any chance to reply.

The main fix moves the eager initialization onto a worker thread with asyncio.to_thread, following the pattern initializer_service already uses for the same reason. The semaphore, the active-task registry and the create_task hand-off all deliberately stay on the server loop: asyncio.run cancels whatever is still pending when it closes its loop, so a background task created inside the worker thread would be destroyed the instant initialization finished. That shape silently cancels every run, and it is specifically avoided here.

Two concurrency-permit leaks are fixed along the way. Both are on paths the previous except Exception: release; raise could not reach:

• A client disconnecting during initialization raises CancelledError, which is a BaseException and so was never caught.
• The missing scenario_result_id check sat outside the try block entirely.

Either one leaked a permit, and three such failures exhausted the concurrency limit and wedged the server for the rest of the session. For the E2E suite that matters, because one session-scoped backend serves every scenario in the run. The permit is now released from a finally block until ownership transfers to the background task, tracked with an explicit release_on_exit flag so it is released exactly once and never twice.

The response is also built before the task hand-off, so a lookup failure can no longer leave a run executing that the caller never received an id for. The active_tasks entry is unwound on that path too.

Finally, the start_scenario_run route docstring said "Returns immediately", which was not true before this change and is still not true after it. It now describes what actually happens.

Part of the v1.1.0 release wave with #2510, #2511 and #2512.

Tests and Documentation

Six new tests in tests/unit/backend/test_scenario_run_service.py:

• test_start_run_keeps_event_loop_responsive counts heartbeats on the loop during a slow start. A blocked loop yields zero.
• test_start_run_background_task_survives_handoff asserts the run actually executes. This is the test that catches the "silently cancels every run" shape.
• test_start_run_releases_semaphore_when_cancelled_during_init covers CancelledError being a BaseException.
• test_start_run_releases_semaphore_when_result_id_missing covers the check that used to sit outside the try.
• test_start_run_cleans_up_when_response_lookup_fails asserts no stranded permit and no stranded active_tasks entry.
• test_start_run_releases_semaphore_exactly_once_on_success guards against the obvious over-correction of double-releasing.

The first two matter as a pair rather than individually: a responsiveness fix that cancels every run would pass the responsiveness test on its own, so the hand-off test is what makes the first one meaningful.

test_start_run_exceeds_concurrent_limit needed a fix. It was passing for the wrong reason: it relied on the event loop never yielding during start, so the mocked runs completed and handed their permits straight back before the limit could ever be reached. Now that start yields, the test holds its background runs open, which is what a real run does.

Ran pytest tests/unit/backend/test_scenario_run_service.py: 70 passed.

Documentation: the start_scenario_run route docstring is corrected in this PR. JupyText was not run and is not applicable: no notebooks or code samples are affected, and the public API is unchanged.

[Richard Lundeen (rlundeen2)]

(GHCP Generated): FYI only - this PR should not wait on the Scenario stack. Stacked PR #2376 introduces FIFO scheduling in ScenarioRunService and queues fully initialized runs, so it overlaps this service. The event-loop responsiveness fix here is still needed. If #2522 merges first, we will rebase the stack and preserve the worker-thread preparation while adapting the semaphore-specific cleanup that FIFO scheduling supersedes.

[Richard Lundeen (rlundeen2)]

(GHCP Generated): This still returns a Scenario after asyncio.run() closes the worker loop. Draining current tasks does not prove that the scenario is independent of that loop because targets and initializers can retain futures, clients, locks, or tasks bound to it. For example, HuggingFaceChatTarget stores load_model_and_tokenizer_task; loads longer than five seconds now fail during start, while resources without a pending task can escape detection. Please keep the initialization loop alive for the scenario lifetime, run the scenario on the same loop, or offload only the blocking work. The earlier loop-lifetime concern is not fully resolved.

[Richard Lundeen (rlundeen2)]

(GHCP Generated): pending is a one-time snapshot. An initialization task can create another task before it finishes. The child is not included in this asyncio.wait, still_running is empty, and asyncio.run() then cancels the child while the scenario returns as healthy. If this temporary-loop design remains, please use one deadline and repeat asyncio.all_tasks() until no other tasks remain.

[hannahwestra25]

we also need the case when a result was cancelled and now i being intentionally resumed so to distinguish its init state from a cancellation during preparation we could do

Suggested change

	if response.status == ScenarioRunState.CANCELLED:
	resume_was_cancelled = False
	if request.scenario_result_id:
	existing = self.get_run(scenario_result_id=request.scenario_result_id)
	resume_was_cancelled = (
	existing is not None
	and existing.status == ScenarioRunState.CANCELLED
	)
	# After preparation
	if (
	response.status == ScenarioRunState.CANCELLED
	and not resume_was_cancelled
	):
	return response

also add test

[hannahwestra25]

There’s still a terminal-state race here. A cancellation can set the row to CANCELLED while preparation is draining, then this update overwrites it with FAILED.

Suggested change

	self._memory.update_scenario_run_state(
	scenario_result_id=scenario_result_id,
	scenario_run_state=ScenarioRunState.FAILED,
	error_message=str(drain_error),
	error_type=type(drain_error).__name__,
	)
	self._memory.try_update_scenario_run_state(
	scenario_result_id=scenario_result_id,
	expected_states={
	ScenarioRunState.CREATED,
	ScenarioRunState.IN_PROGRESS,
	},
	scenario_run_state=ScenarioRunState.FAILED,
	error_message=str(drain_error),
	error_type=type(drain_error).__name__,
	)