Skip to content

ship dev to main: LKG fallback + cache-key purge (prod aggregate poisoned) - #1189

Closed
Flotapponnier wants to merge 1 commit into
mainfrom
ship/dev-to-main-cachefix
Closed

Flotapponnier wants to merge 1 commit into
mainfrom
ship/dev-to-main-cachefix

Conversation

@Flotapponnier

Copy link
Copy Markdown
Collaborator

Prod incident: the all-benchmarks aggregate cache holds 9 draft benches (poisoned during an SRH congestion window at 20:03 UTC) and every revalidation throws AllBenchmarksDraftError, preserving the poison indefinitely. No revalidation route exists; the repo pattern is a cache-key bump + deploy.

Ships dev (all staging-gated benches stay hidden on prod via REMOVED_BENCH_SLUGS) including:

Infra fixes already applied VPS-side: SRH_CONNECTION_POOL_SIZE=25 (root cause), worker on direct TCP.

@Flotapponnier
Flotapponnier force-pushed the ship/dev-to-main-cachefix branch from e7ee27d to bd9184b Compare September 11, 2026 15:57
@Flotapponnier

Copy link
Copy Markdown
Collaborator Author

Closing during a branch cleanup: this has sat open since its creation date and the work has either shipped another way or gone stale. The diff stays readable on this page if any of it is wanted again.

@Flotapponnier
Flotapponnier deleted the ship/dev-to-main-cachefix branch September 22, 2026 18:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant