Field report
"glowscript.org is throwing 500s" — first morning of fall semester, from a classroom. The site was in fact healthy (910 successful sign-ins before 8:40am Pacific); what the classroom saw was the sign-in callback failing with Google's bare "500 Server Error" page.
The mechanism
The authlib OAuth state is single-use and lives in the session cookie. /google/auth therefore 500s with MismatchingStateError for two real populations:
- Anyone who refreshes the callback URL. The state was consumed on the first attempt, so every subsequent load of the same URL can only fail. Observed live: one machine (a classroom podium or projector box, from the IP) retried a single dead callback 68 times, each retry rendering the bare 500 page. From inside that room, the site is down.
- Browsers that refuse the session cookie — no state is ever stored, so the first attempt fails the same way.
How long has this been happening?
As far back as log retention reaches — a steady 1–2% of all sign-ins, at summer traffic (~150/day) and semester traffic (900+/day) alike:
| day |
ok |
500 |
rate |
| Aug 04 |
156 |
2 |
1.3% |
| Aug 18 |
~896 |
4 |
0.4% |
| Aug 25 |
~887 |
13 |
1.4% |
| Aug 28 |
~883 |
17 |
1.9% |
| Sep 01 |
910 |
90 |
9% raw / ~2.4% excluding the one 68-retry machine |
The last deploy was July 15; the rate is unchanged across it. Nothing regressed — semester start just made a chronic failure loud enough to be reported.
The fix
A retry of the same callback URL can never succeed, so the only useful response is a clean landing where the user can start over. Catch OAuthError around authorize_access_token() and redirect to /; same for the bare-/google/auth-with-no-state branch, which currently logs "Yikes!" and then falls through to a guaranteed crash.
PR follows with tests that reproduce the exact production exception.
(Separately observed while diagnosing, not part of this issue: a ~7-minute whole-site outage 15:46–15:53 UTC today caused by App Engine declining to schedule instances for 37 minutes — no quota, billing, or config cause visible, no declared GCP incident. min_instances: 1 would have ridden through it; worth its own discussion.)
Field report
"glowscript.org is throwing 500s" — first morning of fall semester, from a classroom. The site was in fact healthy (910 successful sign-ins before 8:40am Pacific); what the classroom saw was the sign-in callback failing with Google's bare "500 Server Error" page.
The mechanism
The authlib OAuth state is single-use and lives in the session cookie.
/google/auththerefore 500s withMismatchingStateErrorfor two real populations:How long has this been happening?
As far back as log retention reaches — a steady 1–2% of all sign-ins, at summer traffic (~150/day) and semester traffic (900+/day) alike:
The last deploy was July 15; the rate is unchanged across it. Nothing regressed — semester start just made a chronic failure loud enough to be reported.
The fix
A retry of the same callback URL can never succeed, so the only useful response is a clean landing where the user can start over. Catch
OAuthErroraroundauthorize_access_token()and redirect to/; same for the bare-/google/auth-with-no-state branch, which currently logs "Yikes!" and then falls through to a guaranteed crash.PR follows with tests that reproduce the exact production exception.
(Separately observed while diagnosing, not part of this issue: a ~7-minute whole-site outage 15:46–15:53 UTC today caused by App Engine declining to schedule instances for 37 minutes — no quota, billing, or config cause visible, no declared GCP incident.
min_instances: 1would have ridden through it; worth its own discussion.)