Optimize stats api - #465
Merged
Merged
Conversation
duckduckgrayduck
force-pushed
the
stats_api_optimize
branch
from
September 10, 2026 22:27
4ec9775 to
5a4f57b
Compare
eyeseast
approved these changes
Sep 11, 2026
mitchelljkotler
approved these changes
Sep 11, 2026
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Unfortunately I was hitting timeouts on pages with users with large document sets on production. Wasn't experiencing this on staging, but staging is much smaller.
This approach moves to stored fields for total_documents and recent_upload_count for the stats_api. It means we don't have to annotate per page anymore, so that's gone. The loss is we don't get a live statistic for these counts, but considering this gets synced once daily anyway, this tradeoff is necessary. There is a management command (that I tested in staging) that recomputes these two fields and a Celery task that runs this command for us nightly.
The compute queries the Django management command calls I ran in the Django shell on production to test, these are their times:
user totals scan: 59.59 s
user recents scan: 12.2 s
org totals scan: 53.08 s
org recents scan: 0.47 s
I think is acceptable for a nightly scan. Not something acceptable for a web request, but in a Celery task, definitely.
The computed stats approach seems to be no issue on MuckRock's side, so I'm not touching that for now.
aged_out is still necessary for the caller because it won't know when a recompute happens once the boundary is crossed, but I added an optimization there as well to make it cheaper and for it to short circuit per org. This also adds the timezone naiveness check that the MuckRock stats API got in feedback that didn't get caught here.