Scan a large library a day at a time, and choose which episodes to write

README.md

@@ -47,6 +47,11 @@ Intro and Skip Credits buttons appear without any local analysis.
 - **Alternate sources fill the gaps.** Chapter names Plex already extracted, and
   optional local detection, cover items TheIntroDB does not have yet. They can
   never override TheIntroDB for a segment type it answered.
+- **A large library finishes, a day at a time.** An item that has been looked up
+  is remembered, so it is never asked about twice. A run spends the day's
+  request allowance on the items it has never seen, stops when that is used up,
+  and the next run carries on from the same place. A library of tens of thousands
+  of items converges over a few days instead of re-asking forever.
 
 ---
 
@@ -248,7 +253,7 @@ Run `plex-sync` with no arguments in a terminal and you get the interface:
 | --- | --- |
 | `1` | Status: what is in the library, what the ledger recorded, API quota |
 | `2` | Library: every matched item with its ids, sources and marker state |
-| `3` | Plan: exactly what a run would change, before it changes anything |
+| `3` | Plan: exactly what a run would change. Up/down moves, space turns an item on or off, `A`/`N` select all or none, `R` marks one for a re-scan |
 | `4` | Runs: history, and the undo journals from previous applies |
 | `5` | Settings: every setting, editable in place with the arrow keys and enter |
 | `?` | Help |
@@ -266,6 +271,8 @@ plex-sync config check          # validate configuration and reach both services
 plex-sync library               # list items and the names Plex uses, for --show
 plex-sync preview --show "the last of us"   # what a run would change, for one show
 plex-sync preview --save preview.json  # ...and save it, to write later without Plex
+plex-sync preview --deselect 1234      # leave a rating key out of the write
+plex-sync preview --rescan tmdb:1399:1:1   # ask about one item again
 plex-sync apply --yes           # write the markers (--dry-run to see them first)
 plex-sync apply --preview preview.json --yes   # write a saved preview, without Plex
 plex-sync undo latest --yes     # revert the most recent run
@@ -298,6 +305,40 @@ database. If you would rather your own scheduler owned it, `sync --yes` and
 See [docs/scheduling.md](docs/scheduling.md) for systemd, launchd, Task
 Scheduler, cron and container arrangements.
 
+### Large libraries
+
+TheIntroDB allows 1000 requests per UTC day with an API key, and 500 without one.
+A library of tens of thousands of items cannot be scanned in a day, so the tool
+is built to be left running rather than to be finished in one pass:
+
+- **An item is scanned once.** Every lookup is recorded, and a recorded item is
+  answered from the ledger afterwards without a request, however long ago it was
+  scanned. Expiring that record is what would make a large library never finish,
+  so nothing expires it.
+- **A run uses the whole day's allowance.** It scans the items it has never seen
+  until the allowance is spent, then stops cleanly and logs how many are left.
+  That is not an error, and the nightly timer keeps its schedule.
+- **The next run continues.** Because every scan is recorded, it picks up exactly
+  where the last one stopped. A 40,000-item library at 1000 requests a day
+  converges in about 40 days, and each run writes the markers it has data for.
+- **Re-scanning is deliberate.** Nothing is asked about twice on a timer. When a
+  submission lands and you want fresh answers, name the items:
+
+```bash
+# One item, by its lookup key (the same key the Library screen shows):
+plex-sync sync --yes --rescan tmdb:1399:1:1
+
+# Everything, at the cost of a full library of requests:
+plex-sync sync --yes --rescan-all
+```
+
+In the interface, press `p` for a plan, move with the arrow keys, and press `R`
+on the item you want re-scanned; planning again is what asks for it. `space`
+turns an item off, which leaves it in the plan but out of the write.
+
+`plex-sync status` reports how many items have been scanned and how they split,
+which is the number that says how far through the library the tool has got.
+
 ---
 
 ## Configuration

docs/architecture.md

@@ -71,8 +71,8 @@ expects, and the safety rules that come with editing a live database.
 | `internal/model` | the shared data types | nothing |
 | `internal/plexapi` | enumerate libraries, read ids, chapters and existing markers | Plex HTTP |
 | `internal/plexdb` | read and write markers, back up, undo | Plex SQLite |
-| `internal/tidb` | TheIntroDB client: pacing, budget, cache, null times | TheIntroDB HTTP |
-| `internal/ledger` | durable state: lookup cache, what we wrote, run history | our SQLite |
+| `internal/tidb` | TheIntroDB client: pacing, budget, scan records, null times | TheIntroDB HTTP |
+| `internal/ledger` | durable state: scan records, lookup cache, what we wrote, run history | our SQLite |
 | `internal/planner` | merge sources, map types, resolve ranges, decide the change set | nothing |
 | `internal/source` | markers from chapter names Plex extracted, and local detection for the rest | `plexapi` output, ffmpeg + fpcalc |
 | `internal/schedule` | the cron subset the process holds its own timer with | nothing |
@@ -111,6 +111,37 @@ A segment type is won by the first enabled source that has it, and each written
 marker records which source produced it, so the status page and the plan output
 can always answer "where did this timing come from".
 
+## Scanning a large library
+
+Two different questions get answered by two different pieces of ledger state,
+and keeping them apart is what makes a library of tens of thousands of items
+finish:
+
+- **"Do we still trust this body?"** — the lookup cache (`lookups`), with a TTL.
+  A 200 is cached for `theintrodb.hit_ttl_days` and a 404 for
+  `theintrodb.miss_ttl_days`, because a 404 becomes a 200 the moment someone
+  submits the timing.
+- **"Have we spent a request on this item at all?"** — the scan record
+  (`scans`), with no expiry. A record means the item has been looked up, and a
+  lookup that finds one is answered from the cache without a request.
+
+The second is what a large library runs on. Time alone never causes a request:
+`LookupForced` is the only thing that asks about a scanned item again, and that
+is reached by naming an item for a re-scan (`--rescan`, `--rescan-all`, or `R`
+on the Preview screen). Expiring the scan record would put the library back to
+re-asking it forever and never finishing.
+
+When the day's allowance is spent the client refuses the next request, and
+`internal/sync` treats that as a stopping point rather than an error: the run
+ends, records the number of items still to scan, and the next run begins where
+it left off. The nightly timer therefore makes progress every night instead of
+failing every night.
+
+A plan also carries a `Selection`: the items turned off, and the items named for
+a re-scan. It is recorded in the plan file so a plan made on a host and applied
+in a container writes the same subset, and so the interface's selection is a
+decision the writer honours rather than a note the writer ignores.
+
 ## Idempotence and provenance
 
 The ledger records every marker the tool wrote. On the next run, a marker that

docs/scheduling.md

@@ -19,6 +19,37 @@ writes without `--yes`, and nothing writes while someone is watching something.
 Every run writes a line per item to the ledger and a log line to standard error,
 so `journalctl`, the Docker log or a file all work as a record of what happened.
 
+## Large libraries, and the daily allowance
+
+TheIntroDB allows 1000 requests per UTC day with an API key and 500 without one,
+so a library of tens of thousands of items is not scanned in one night. That is
+expected, and the schedule is how it is meant to be worked through:
+
+- An item that has been looked up is recorded, and a recorded item is answered
+  from the ledger afterwards without a request. Nothing expires that record, so
+  an item is asked about once rather than once every couple of weeks.
+- A run spends what is left of the day's allowance on items it has never seen,
+  then stops. It logs how many items are still to scan and exits **zero**: the
+  allowance being spent is a stopping point, not a failure, and a nightly timer
+  that reported an error every night would be a timer nobody reads.
+- The next run continues from the same place, because every scan is recorded.
+
+A schedule therefore converges: a 40,000-item library at 1000 requests a day
+gets through in about 40 days, writing the markers it has data for each night.
+Raising `theintrodb.daily_budget` beyond the allowance does not help — the
+server refuses the request either way; the budget is what stops the client
+before the refusal.
+
+To ask about something again, name it. Nothing is re-scanned on a timer:
+
+```bash
+plex-sync sync --yes --rescan tmdb:1399:1:1    # one item, by lookup key
+plex-sync sync --yes --rescan-all              # everything; a full library of requests
+```
+
+`plex-sync status` reports how many items have been scanned, split by whether
+TheIntroDB had data, which is the progress figure for a library this size.
+
 ## The built-in schedule
 
 ```bash
@@ -201,8 +232,10 @@ task forever.
 30 7 * * *  PLEX_SYNC_STATE_DIR=/var/lib/plex-sync /usr/local/bin/plex-sync schedule --once --yes >>/var/log/plex-sync.log 2>&1
 ```
 
-A run that finds nothing to do is not an error. One that fails exits non-zero
-and logs why; cron will mail that to you if the machine can send mail.
+A run that finds nothing to do is not an error. One that has simply used up the
+day's request allowance exits zero as well — see "Large libraries" above — while
+one that genuinely fails exits non-zero and logs why; cron will mail that to you
+if the machine can send mail.
 
 ## Docker
 

internal/api/server.go

@@ -303,8 +303,9 @@ func (s *Server) register() {
 		Path:        "/plan",
 		Summary:     "Compute the change set without writing anything",
 		Description: "Asks TheIntroDB about every item, which can take minutes on a large " +
-			"library. Every answer is cached in the ledger, so a following apply does not " +
-			"pay for the requests twice.",
+			"library. An item that has already been scanned is answered from the ledger without " +
+			"a request, so a following apply does not pay for the requests twice and a large " +
+			"library is worked through a day at a time.",
 		Tags: []string{"run"},
 	}, func(ctx context.Context, in *planInput) (*planOutput, error) {
 		res, err := s.runner.Plan(ctx, sync.Options{

internal/cli/commands.go

@@ -29,6 +29,14 @@ func addRunFlags(cmd *cobra.Command, opts *sync.Options) {
 	flags.IntSliceVar(&opts.Sections, "section", nil, "limit to these Plex library section keys")
 	flags.StringVar(&opts.Filter, "show", "", "only items whose title contains this text")
 	flags.IntVar(&opts.Limit, "limit", 0, "examine at most this many items")
+	flags.StringSliceVar(&opts.Rescan, "rescan", nil,
+		"ask TheIntroDB about this item again, by lookup key (tmdb:1234:2:5); repeatable")
+	flags.BoolVar(&opts.RescanAll, "rescan-all", false,
+		"ask about every item again, even ones already scanned (costs a full library of requests)")
+	flags.IntSliceVar(&opts.Only, "select", nil,
+		"write only these Plex rating keys; repeatable")
+	flags.IntSliceVar(&opts.Deselected, "deselect", nil,
+		"leave these Plex rating keys out of the write; repeatable")
 	flags.BoolVar(&opts.DryRun, "dry-run", false, "report what would happen without writing")
 	flags.BoolVar(&opts.PlexStopped, "plex-stopped", false, "assert that Plex is stopped")
 	flags.BoolVar(&opts.SkipSessionCheck, "skip-session-check", false, "skip the active session check")
@@ -315,9 +323,14 @@ writes nothing and exits non-zero rather than doing something surprising.`),
 				return encoder.Encode(res)
 			}
 			out := stdout(cmd)
-			fmt.Fprintf(out, "examined %d item(s), %d with data, %d without, %d lookup(s), %d cached\n",
+			fmt.Fprintf(out, "examined %d item(s), %d with data, %d without, %d already scanned, %d lookup(s), %d cached\n",
 				res.Survey.Items, res.Survey.WithData, res.Survey.NoData,
-				res.Survey.Lookups, res.Survey.Cached)
+				res.Survey.Skipped, res.Survey.Lookups, res.Survey.Cached)
+			if res.Survey.Paused {
+				fmt.Fprintf(out,
+					"paused: the day's request allowance is spent; %d item(s) still to scan, "+
+						"the next run continues from here\n", res.Survey.Remaining)
+			}
 			if res.Applied {
 				fmt.Fprintf(out, "wrote %d marker(s) across %d item(s), removed %d, skipped %d\n",
 					res.Stats.Added, res.Stats.Written, res.Stats.Removed, res.Stats.Skipped)
@@ -434,6 +447,8 @@ func newStatusCmd(g *globals) *cobra.Command {
 			fmt.Fprintf(out, "ledger            %s\n", stats.DatabasePath)
 			fmt.Fprintf(out, "lookups           %d (%d hits, %d misses)\n",
 				stats.Lookups, stats.LookupHits, stats.LookupMisses)
+			fmt.Fprintf(out, "scanned           %d item(s) (%d with data, %d without); these are never asked about again unless a re-scan is asked for\n",
+				stats.Scanned, stats.ScannedWithData, stats.ScannedNoData)
 			fmt.Fprintf(out, "markers recorded  %d across %d item(s)\n",
 				stats.AppliedMarkers, stats.AppliedItems)
 			fmt.Fprintf(out, "requests today    %d of %d",
@@ -468,8 +483,13 @@ func newStatusCmd(g *globals) *cobra.Command {
 
 func printPlan(cmd *cobra.Command, res *sync.Result, show int) {
 	out := stdout(cmd)
-	fmt.Fprintf(out, "would change %d item(s) from %d examined: %d with data, %d without\n",
-		res.Survey.Planned, res.Survey.Items, res.Survey.WithData, res.Survey.NoData)
+	fmt.Fprintf(out, "would change %d item(s) from %d examined: %d with data, %d without, %d already scanned\n",
+		res.Survey.Planned, res.Survey.Items, res.Survey.WithData, res.Survey.NoData, res.Survey.Skipped)
+	if res.Survey.Paused {
+		fmt.Fprintf(out,
+			"paused: the day's request allowance is spent; %d item(s) still to scan, "+
+				"the next run continues from here\n", res.Survey.Remaining)
+	}
 
 	reasons := make([]string, 0, len(res.Survey.SkipReasons))
 	for reason := range res.Survey.SkipReasons {

internal/cli/schedule.go

@@ -141,6 +141,15 @@ writes with --yes.
 					"markers_removed", result.Stats.Removed,
 					"applied", result.Applied,
 					"took", time.Since(started).Round(time.Second).String())
+				if result.Survey.Paused {
+					// Not a failure: the allowance is spent, everything the
+					// run did is recorded, and the next firing carries on from
+					// the same place.
+					log.Info("the run stopped early: the day's request allowance is spent",
+						"scanned", result.Survey.Skipped,
+						"remaining", result.Survey.Remaining,
+						"resume", "the next run continues from here")
+				}
 			}
 
 			if once {

internal/config/config.go

@@ -84,9 +84,14 @@ type TheIntroDB struct {
 	// pacing at 25 leaves room for the request that lands while one is in flight.
 	MaxPerWindow int     `toml:"max_per_window"`
 	WindowS      float64 `toml:"window_s"`
-	// MissTTLDays is how long a cached "no data" answer is trusted.
+	// MissTTLDays is how long a cached "no data" body is trusted. It expires the
+	// body, not the record that the item was scanned: a 404 becomes a 200 the
+	// moment someone submits the timing, but asking about the item again is a
+	// re-scan rather than something time does on its own.
 	MissTTLDays int `toml:"miss_ttl_days"`
-	// HitTTLDays is how long a cached answer is trusted before a refresh.
+	// HitTTLDays is how long a cached answer body is trusted. As with the miss
+	// TTL, this no longer decides when a request is made: an item that has been
+	// scanned at all is answered from the ledger whatever this says.
 	HitTTLDays int `toml:"hit_ttl_days"`
 }