[fylorn]

Summary

mcp_store_concurrency::parallel_installs_same_template_keep_count_in_sync failed intermittently on dev with 500 Internal Server Error and an empty body.

Root cause

Not the install transaction (advisory lock, FOR UPDATE on the template row and install_count = install_count + 1 are all correct, and Postgres logged no error at the time of either failure). The 500 came from the console auth middleware:

1. The test harness ground each login's proof of work (difficulty 19, about 500k SHA-256 tries on average, with a long geometric tail) inline on the #[tokio::test] current-thread runtime that also drives the in-process server.
2. The test logged in session k+1 while install k was already in flight, so a slow grind froze that install mid-request.
3. Its connection-pool acquire then outlived sqlx's 5 s acquire_timeout, even though the pool held idle connections. With temporary logging, every reproduced failure showed pool timed out while waiting for an open connection with pool_size=4..5, pool_idle=2..3 out of 20.
4. require_auth mapped that error to a bare StatusCode::INTERNAL_SERVER_ERROR. The permission lookup did this without any log line, which is why CI showed only request failed ... 500 latency_ms=5004.

Changes

• Product (auth_guard.rs, api_key_auth.rs): database errors in the session check, the console API-key path and the gateway API-key path now use the same From<sqlx::Error> for AppError mapping as handlers. A pool timeout is a 503 with the standard JSON error body; any other database error is a 500 with one. Each is logged with the name of the lookup that failed. Unit tests cover both mappings.
• Harness (test-support/src/client.rs): the proof of work is ground on spawn_blocking, so the server keeps serving while a test client logs in. This affects every test that logs in while requests are in flight.
• Test (mcp_store_concurrency.rs): all sessions log in before any install fires, so the five installs actually race each other instead of trickling out one login apart.
• CHANGELOG [Unreleased] Fixed entry.

Reproduction (local, Postgres 18 + Redis 8 in dedicated containers, CPU saturated with busy loops to mimic a loaded runner)

Build	Runs	Failures
dev (before)	25	4 (three empty 500s from require_auth, one 503 from the handler's own pool acquire)
harness fix only, original test order	25	0
full fix	30	0

With the machine idle, both before and after pass 10/10. Locally: clippy -D warnings (lib/bins and tests), fmt and nextest --lib --bins (840 passed) all pass. The integration suite run without ClickHouse gave 386 passed; all 72 failures were ClickHouse-database creation.

🤖 Generated with Claude Code