Validate map tile archives before extracting and never publish a partial file

[dirkpetersen]

Fixes a data-corruption path in the offline tile download: a truncated or interrupted download could end up as a corrupt tile at a live path, after which mapd logged could not unmarshal offline data: unexpected EOF for that area on every load and could not repair it by downloading again.

What happened

On a drive with the full WA/OR/ID region on disk (~286 MB), the device started re-downloading the whole region over cellular. maps.ReadOffline was failing with unexpected EOF on several tiles that were shorter than they should have been. Tracing it back, the per-archive block in settings/download.go has four independent ways of leaving corrupt bytes at a live tile path:

1. No validation before extraction. The archive was extracted straight over existing tiles, so a truncated download was only discovered at read time, after it had already overwritten good data.
2. Download written directly to its final path (os.Create). A process killed mid-transfer (power cut, reboot, OOM) leaves a short file that looks complete.
3. Extraction opened targets with O_CREATE|O_RDWR and no O_TRUNC. Writing a shorter tile over a longer one left the old tail attached — a hybrid that survived re-downloading, which is why fetching the region again did not fix it.
4. Every extraction error was logged and ignored (a failed os.Open handed a nil file to gzip.NewReader, a failed gzip parse handed a nil reader to tar.NewReader, a failed io.Copy left a half-written file), and the tile was still counted as downloaded.

The change

All in settings/download.go; no schema, settings, CLI or message changes.

• DownloadFile writes to <dest>.part and renames into place only after the body is complete (and matches Content-Length when the server declares one). Any failure removes the .part. The destination either does not exist or is complete.
• ValidateArchive reads the whole gzip+tar stream to io.Discard before anything is extracted. This has to drain the gzip stream, not just walk the tar: tar.Next returns io.EOF at the tar end-of-archive marker, which sits before gzip's CRC32/ISIZE trailer, so a tail truncation of a few bytes validates clean if you stop at the tar level. (My first version had exactly that bug; the 99 %-truncation test case caught it.)
• extractArchive replaces the inline loop and returns an error on the first failure instead of warning and continuing. Each regular file is written through writeFileAtomic (.part + Sync + Rename), which also fixes the stale-tail bug — the rename replaces the file wholesale, so length cannot carry over.
• In downloadBounds, the per-archive flow is now download → validate → extract, and a tile that fails validation or extraction is discarded and not counted in DownloadedFiles, the same way a failed HTTP fetch already isn't.
• safeJoin rejects archive entries that would escape the destination (../). The tile host is trusted, so this is hardening rather than an observed problem — happy to drop it if you'd rather keep the PR to the bug.

Tests

settings/download_validate_test.go (new; the package had no tests): intact archive accepted; truncation at 10/50/90/99 % rejected; garbage, empty and no-file archives rejected; DownloadFile publishes a complete file and no .part, rejects a short body against Content-Length, does not publish on HTTP error, ignores a stale .part, and — asserted from inside the test HTTP handler while the copy is in flight — the destination does not exist mid-transfer; writeFileAtomic replaces a longer file wholesale and leaves no .part; safeJoin rejects .. entries and allows normal ones; and a truncated archive leaves an existing live tile untouched.

Two of the guards are deliberately weaker claims than they look, checked by mutation: Go's HTTP client already reports a short body against a declared Content-Length as io.ErrUnexpectedEOF, so the explicit length check is defence-in-depth; and O_TRUNC on the .part is belt-and-braces — the rename is what fixes the stale tail.

Also checked against real archives from map-data.pfeifer.dev, on this exact branch: offline/46/-124.tar.gz (35.3 MB) and offline/46/-122.tar.gz (8.6 MB) both validate, and each minus its last 3 bytes is rejected with unexpected EOF.

go build, go vet ./..., go test ./... and gofumpt -l are clean. A native-arm64 build with this change has been running on a comma 3X since 2026-09-03 with a full region on disk; the device has not lost map data since.

Compatibility

• Behaviour on the success path is unchanged apart from the extra validation pass (each archive is decompressed twice: once to validate, once to extract). On the two real archives above that is ~0.3 s for the largest cell on x86, well under the transfer time for the same file.
• On the failure path the tile is now skipped and logged instead of being written corrupt; DownloadedFiles for that location ends short of TotalFiles, which is what a failed fetch already does today. No new progress fields.
• A crash mid-extract can leave a stray <tile>.part next to a live tile. The reader opens tiles by exact path (FindWaysAroundPosition → os.ReadFile), so it is inert, and the next extract of that tile removes it.
• Does not conflict with Reduce non-intersecting map archive downloads #136: that PR restructures the loop around the per-archive block and leaves the block itself as-is, so the two apply cleanly in either order.
• Does not change what happens when a tile that is already corrupt on disk is read — ReadOffline still returns Loaded: false without deleting it. Deleting an unreadable tile so it gets re-fetched would be a sensible follow-up; left out to keep this to the download path.

🤖 Generated with Claude Code

https://claude.ai/code/session_01FwdRXRST5H2Hoft9yDkyU3

[dirkpetersen]

I drove a few weeks with this validation feature and when i enable automated real time downloading ..... now it no longer attempts to download maps multiple times for me.