Skip to content

fix(storage): resume history maintenance and unblock full backups - #1340

Closed
frahlg wants to merge 5 commits into
masterfrom
fix/history-bounded-maintenance-20260921
Closed

frahlg wants to merge 5 commits into
masterfrom
fix/history-bounded-maintenance-20260921

Conversation

@frahlg

@frahlg frahlg commented Sep 21, 2026

Copy link
Copy Markdown
Member

Large histories made maintenance repeatedly scan the same data, restart interrupted archive work and hold full backups behind a non-cancellable mutex. On the affected Raspberry Pi, dashboard retention timed out while live commits continued and the 6.7 GB history database retained roughly 161 million old samples.

Archive jobs now commit their input/prune cursor with bounded scratch batches, resume after cancellation or restart, and yield between stages. Publication still requires full readback verification; resumed pruning checks the published file hash and matches source values. Timestamp and staging-key seeks avoid repeated scans, and dashboard pruning uses its existing expiry index. Legacy archive sealing retries reader contention and rechecks late input outside the short write budget. Full backup pauses maintenance, honours cancellation through copying and verification, and removes the fixed per-batch delay that alone exceeded its deadline on this dataset. API/UI report actual processed records, phases and failures.

This keeps the current SQLite and Parquet design and aggregate retention policy: recent detail, minute summaries for 30 days, then five-minute summaries through the existing retention period. FTW is a dependable local EMS record; indefinite raw history belongs in an optional external system. No database engine, retention setting or TSDB service is added.

Validation:

  • make verify passed; all 643 web tests passed. CI on cc76839d is green: 13 successful checks, three scoped checks skipped.
  • Targeted race tests passed for archive resume, replacement/corruption safety, maintenance pause, live writes and backup cancellation.
  • Disposable copy of the affected dataset: minute backlog completed in 14.75 seconds; raw archival progressed through several days and yielded with a durable cursor. One completed day matched all 1,212,323 original key/value records exactly.
  • ARM64 tests on homelab-rpi passed, including interruption/restart and full backup/create/verify/restore using separate fixture data.
  • Rendered progress, backup and failure states in a browser. Human UI review remains before merge.

The local pilot package preserves the already installed charging-evidence work from #1337. This PR targets master and does not include that independent work. It does not merge or publish a release. Live pilot evidence (21 September):

  • Complete local package v0.131.0-beta.1+history.cc76839d, integration commit ea5891ce; original package remains available for rollback.
  • A real backup paused active maintenance, reported advancing copy counts and allowed live commits. Cancelling the request stopped copying and resumed maintenance. This was an intentional cancellation test, not a completed full-site backup.
  • Restart retained the sample cursor at 51,200 copied rows; it later advanced past 772,096 without recopying the existing 1,215,239-row file.
  • The longer run exposed legacy sealing failure under a reader lock. The added regression fails before the fix and passes after it, including with the race detector.
  • Final package completed multiple maintenance turns with zero maintenance failures, rejected ticks or commit failures. The first large sample file finished full publication/readback, then resumed verified source pruning past 214,016 rows; legacy compaction advanced through multiple older days. Three drivers were healthy, the existing planner mode had a fresh plan, and identity/configuration hashes matched.
  • The old backlog is still being processed. The 6.7 GB SQLite file has not been vacuumed or claimed as reclaimed disk space. The original off-box snapshot remains intact and its SHA-256 was checked again before installation.

Homelab night, 21–22 September

The pilot kept history.db at 6,731,206,656 bytes. Five maintenance turns hit context deadline exceeded and then waited an hour, because the stage budget was cancelled and replaced by the live parent context before the result was read. The rolloff loop treats failed as the hourly schedule and pending as five seconds.

d2f98e50 reads the stage context before that replacement. A spent dashboard or archive budget stays pending and the loop resumes. A dashboard bucket that cannot commit inside its own write budget still fails. Checked with TestHistoryMaintenanceBudgetStaysPending and the existing archive, health and rolloff tests. The running homelab package is still cc76839d.

A stage timeout was read after its context had been replaced by the live parent, so the rolloff loop stored a failure and waited an hour.
The archive turn only checked the clock between batches, so one slow read could keep the card busy until it finished. The stage context now ends with the budget, and the hourly rollup waits for the history writer only inside that budget.
@frahlg

frahlg commented Sep 22, 2026

Copy link
Copy Markdown
Member Author

Superseded by #1341, which is merged. The live path keeps the ledger and chart buckets in SQLite and does not archive raw polls.

@frahlg frahlg closed this Sep 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant