Scan a large library a day at a time, and choose which episodes to write - #1
Merged
Pasithea0 merged 3 commits intoSep 23, 2026
Merged
Conversation
… item A library of tens of thousands of items cannot be looked up against a daily allowance of 500 or 1000 requests, and the lookup cache was not helping: a 14-day miss TTL and a 30-day hit TTL mean every item comes back round for another request eventually, so a large library never converges. Reported as "the cache is good for 404s, but doesn't help with large libraries". Separate the two questions the ledger was answering with one piece of state: - whether a cached body is still trustworthy, which the lookup cache with its TTLs answers, and - whether the item has been asked about at all, which a new `scans` table answers and which never expires. A lookup that finds a scan record is answered from the ledger with no request, however long ago it was scanned. The only thing that asks about a scanned item again is `LookupForced`, reached by naming the item for a re-scan: --rescan, --rescan-all, or a plan that carries the key in its selection. When the day's allowance is spent the client refuses the next request, and a plan now treats that as a stopping point rather than a failure: the run ends, records how many items are still to scan, and is recorded as `paused` rather than as an error, so the nightly timer makes progress every night instead of failing every night. The next run continues where the last one stopped, which is what a 40,000-item library at 1000 requests a day needs. A plan also carries a `Selection` now: which rating keys were turned off, and which lookup keys were named for a re-scan. It travels with the plan file, so a plan made on a host and applied in a container writes the same subset, and the writer honours the interface's selection instead of ignoring it. The schema version advances to 2; opening an older ledger adds the table and records the version, since every migration statement is idempotent.
The preview screen listed what a run would change but gave no way to act on it: the arrow keys scrolled, and the whole plan was written whatever was on screen. Give it a cursor. Up and down move over the items, space turns the one under the cursor on or off, A selects everything and N none, and an item that is off stays in the preview and is left out of the write, so nothing is hidden. R marks the item under the cursor for a re-scan; planning again is what asks TheIntroDB about it, which is how an item that was scanned months ago is refreshed without anything re-scanning on a timer. The selection is applied when the plan is written and is counted in the confirmation, so "Write N markers across M items" is what will actually happen rather than what the plan found.
Add a "Large libraries" section to the README and docs/scheduling.md, and a "Scanning a large library" section to docs/architecture.md, covering the thing that was missing: an item is scanned once, a run uses the whole day's allowance on items it has never seen, a run that stops early is not a failure, and asking about something again is deliberate (--rescan, --rescan-all, or R on the preview screen). The TTL knobs stay, described honestly: they expire a cached body, and no longer decide when a request is made.
Pasithea0
deleted the
feat/large-library-scanning-and-episode-selection
branch
September 23, 2026 20:54
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two problems with a companion app run against a real library: the cron re-asks
about items it already knows, so a large library never converges, and the
preview screen shows what would change but gives you no way to decide.
What was wrong
The ledger cached answers with a TTL — 14 days for a 404, 30 for a 200 — and the
TTL was what decided when a request was made. On a library of tens of thousands
of items against a daily allowance of 500 (or 1000 with a key) that means the
library is re-covered forever and never finishes: today's 500 requests go on
items whose entries happened to expire, not on the items that have never been
asked about.
What this does
Scan records. A new
scanstable records that an item has been looked up, andwhat came back. It never expires. A lookup that finds one is answered from the
ledger with no request, however long ago it was scanned; the cache TTLs still
expire the stored body, but they no longer decide when a request happens. The
404 cache is kept — it is the right thing for a 404, which becomes a 200 the
moment someone submits.
Progressive daily scanning. When the day's allowance is spent, the client
refuses the next request and a plan treats that as a stopping point: the run
ends, logs how many items are still to scan, and is recorded as
pausedratherthan as an error. The next run continues where the last one stopped. A run is no
longer a failure for having run out of requests, so a nightly timer makes
progress every night instead of reporting an error every night. A 40,000-item
library at 1000 requests a day converges in about 40 days, writing the markers it
has data for each run.
Re-scanning is deliberate. Nothing is asked about twice on a timer. The only
thing that re-asks is
LookupForced, reached by naming the item:--rescan <lookup-key>,--rescan-all, or a plan that carries the key.Episode selection on the preview screen. Up/down move a cursor, space turns
the item under it on or off,
A/Nselect all or none,Rmarks one for are-scan (planning again is what asks). An item that is off stays visible in the
plan and is left out of the write. The plan carries a
Selection(unselectedrating keys, and the keys named for a re-scan) so it travels with the plan file
and a container applying a host-made plan writes the same subset.
New flags on
preview,applyandsync/schedule:--rescan,--rescan-all,--select,--deselect.statusreports how many items havebeen scanned and how they split, which is the progress figure for a library this
size.
Safety
Nothing about the write path changed: still
--yes, still a preflight thatrefuses while Plex is running unannounced or something is streaming, still a
backup and an undo journal. The ledger schema goes to version 2; opening an older
ledger adds the table and advances the recorded version, and every migration
statement is idempotent.
Tests
go build ./...,go vet ./...,gofumpt -l .clean;go test -count=1 ./...green. New coverage: ledger scan-record lifecycle and the older-ledger upgrade,
the permanent-skip and forced re-scan behaviour at the client, the plan
selection and the sync-side filter, and the preview screen's cursor, selection,
re-scan marking and rendering. The three tests that asserted the old TTL
re-fetch behaviour now assert what replaced it.