Skip to content

SD card and CLI rough edges (EPIC-15) - #16

Merged
diegoparrilla merged 4 commits into
release/v1.1.0from
epic/15-sd-and-cli-rough-edges
Sep 17, 2026
Merged

diegoparrilla merged 4 commits into
release/v1.1.0from
epic/15-sd-and-cli-rough-edges

Conversation

@diegoparrilla

Copy link
Copy Markdown
Contributor

A reinserted SD card works again without a reset, the device says when there
isn't one instead of failing six different ways, a retried write chunk is no
longer written twice, and the CLI reports a dropped connection instead of a
Python traceback.

A reinserted card stayed dead — two causes, not one

The driver never learns the card left. sd_card_spi_init() begins with "Check if we're not
already initialized before proceeding"
and returns early unless STA_NOINIT is set — and with
card-detect disabled on this board nothing ever sets it when a card is pulled. So on reinsertion the
driver still believed the old card was initialised, skipped its entire init sequence, and handed
FatFs a freshly powered card it had never spoken to, which failed on the first sector read.
sd_card_p->deinit() sets the bit back.

FatFs never asks. While the volume stays registered, mount_volume() never calls
disk_initialize() at all, so f_mount(NULL, "0:", 0) has to come first.

Both are needed. The first version of the fix had only the unregister, and the hardware caught it:
every retry still failed with FR_DISK_ERR.

And the removal cannot be detected from FatFs results. With the card physically out, volume
answered 200 with the cached size and free space, and a listing answered 200 with an empty
directory — FatFs serves what it cached and reports no error, so the "remount on the next failure"
trigger in the plan could never fire. The main loop now asks the card directly every 2 s by reading
sector 0 through disk_read(), which a cache cannot answer.

No card: one answer, and the menu says so

With no card Before After
volume 200 + cached free space 503 no_sd_card
listing 200 + empty directory 503 no_sd_card
download 500 disk_error 503 no_sd_card
upload / mkdir / runner load 500 503 no_sd_card

The menu header carries the state (GEMDRIVE SD: mounted / SD: NO CARD) and [G] and [U] are
refused with "Insert a working microSD card: GEMDRIVE needs one." rather than launching a mode
with no drive behind it. STORY-01's retry loop clears the block about two seconds after a card is
inserted, with no reset.

The state went on the header line rather than a row of its own because a new row breaks the
menu
: the dividers are at fixed pixel rows and the icons are right-aligned per header row, so one
extra line struck Screenmem and Hook through, collided IP address with the USB CDC heading and
ran the status line into Select an option:. A framebuffer snapshot caught that.

A retried write chunk was written twice

After a timeout the ST re-sends the same 1 KB chunk with a fresh token, and the RP appended it
again: the file gained a duplicate and lost its tail. Nothing on the wire distinguished a retry from
new data.

An absolute offset does not work without more state than it looks: the m68k knows only how far it
has got within the current Fwrite call, not the file position, which the RP owns and Fseek
moves — so a relative offset cannot tell a retry of a call's first chunk from the first chunk of the
next call. Instead the m68k carries a sequence number that never repeats: write_seq, bumped once
per chunk and never per retry, sent in d5, which the protocol always transmits and the RP was
skipping. Each open file slot remembers the last sequence it accepted and the bytes it wrote; a
repeat is answered with that same count and not written again.

The CLI no longer prints tracebacks

urllib wraps some connection failures in URLError and lets the rest through as bare OSErrors.
Two exposures: urlopen() itself, and the reads after a successful open — the download loop and
the debug tail loop both called resp.read() with no handler at all, which is where a reboot
mid-transfer lands. CONNECTION_DROPPED names the family once (including IncompleteRead), and
everything maps to one error: line and exit 8.

Verified on hardware

Diego at the Atari, pulling the card and running the copies:

  • Card pull: detected in ~2 s (sector read failed -- card gone), remounted ~4 s after
    reinsertion, volume and listings correct, 256 KB upload round-tripping identical, no reset.
  • Booted with no card: retries every 2 s and picks one up when it appears.
  • No-card gate: menu shows SD: NO CARD, [G] refused, all endpoints 503 no_sd_card; block
    lifts by itself on reinsertion.
  • Retried write: a 512 KB copy with two chunks stalled 10 s each — the ST re-sent both, the RP
    logged repeat of chunk N, answering 1024 again for each, and the copy is byte-identical to
    the source. A first attempt at 2 s stalls produced no retries at all (inside the ST's
    COMMAND_TIMEOUT) and would have been a false pass.
  • CLI against a real device: reset during a 4 MB download → connection dropped by the device (timed out), exit 8; panic during a 4 MB upload → [Errno 32] Broken pipe, exit 8.
  • smoke.py 8/8 debug; CLI suite 119 tests (4 added), and the new tests were checked against the
    unfixed CLI, where all four fail with the traceback this was filed for.

Not verified

  • Pulling the card while the ST has GEMDRIVE open. Today's pulls were with the ST idle or at the
    menu; the interesting case holds open handles across the pull.
  • During a 4 MB upload, three 800 ms SELECT presses were not seen by the firmware, though the
    same injection works on an idle device. This could be the main loop not polling under load or the
    SWD injection failing to hold the pad; they need different fixes, so it is recorded rather than
    guessed at. A physical press during an upload would settle it.

Also found, not fixed here

Changing [D]rive in the menu does not change the drive the ST uses: the letter reaches the m68k in
the reply to its HELLO, which only happens when the ST boots the cartridge. The menu showed K:
while the console kept reporting drive=C (#2) until the ST was reset. Worth its own story.

The cartridge image is now 10,144 of 10,240 bytes.

diegoparrilla and others added 4 commits September 17, 2026 15:38
The card was mounted once at boot, so pulling it and putting it back left
GEMDRIVE and the file API dead until someone reset the device. Two separate
things kept it that way.

The driver never learns the card left. sd_card_spi_init() returns
immediately unless STA_NOINIT is set, and with card-detect disabled on this
board nothing ever sets it when a card is pulled -- so on reinsertion the
driver still believed the old card was initialised, skipped its whole init
sequence, and handed FatFs a freshly powered card it had never spoken to.
sd_card_p->deinit() sets the bit back. And while the volume stays
registered, mount_volume() never calls disk_initialize() at all, so
f_mount(NULL, "0:", 0) has to come first. Both are needed: the first version
of this fix had only the unregister, and every retry still failed with
FR_DISK_ERR until the deinit was added.

Detecting the removal cannot be left to FatFs results either. With the card
physically out, volume answered 200 from the cached free space and a listing
answered 200 with an empty directory -- FatFs serves what it cached and
reports no error, so no failure ever arrives. The main loop now asks the
card directly every 2 s by reading sector 0 through disk_read(), which the
cache cannot answer. FatFs medium errors still mark the card gone, from the
HTTP path and all 15 GEMDRIVE call sites, but the probe is what catches a
pull.

Verified with the card pulled and reinserted: detected in about 2 s,
remounted about 4 s after reinsertion, volume and listings correct again, a
256 KB upload round-tripping identical, no reset and crash count 0. A device
booted with no card also retries every 2 s and picks one up when it appears.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016emgLTZm7w3qZsRbukWwE4
…15 STORY-02)

With no card the device was silently broken: the setup menu said nothing,
[G] and [U] launched a mode with no drive behind it, and the API answered
six different ways -- including two that lied. /volume answered 200 with the
cached size and free space, and a listing answered 200 with an empty
directory, because FatFs serves what it cached.

Every endpoint that needs the card now answers 503 no_sd_card: volume,
listings, downloads, uploads, folder operations, runner load and runner run,
through one guard in route() and one write_no_card(). write_fs_error()
reports a missing card as such rather than as a disk fault. The CLI needed
no change, since it maps by HTTP status and 503 already exits 6.

The menu header carries the state -- GEMDRIVE   SD: mounted / SD: NO CARD --
and emul_canLaunch() refuses [G] and [U] while no card is mounted, with
"Insert a working microSD card: GEMDRIVE needs one." on the status line.
STORY-01's retry loop is what clears it: inserting a card lifts the block
within about two seconds, with no reset.

The state went on the header line rather than a row of its own because a new
row breaks the menu: dividers are drawn at fixed pixel rows and the status
icons are right-aligned per header row, so one extra line struck Screenmem
and Hook through, collided IP address with the USB CDC heading, and ran the
status line into Select an option. The framebuffer snapshot caught it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016emgLTZm7w3qZsRbukWwE4
The device dropping a connection is routine -- a watchdog reboot, a panic or
a SELECT press all cut the socket -- and the user got a Python traceback
ending in ConnectionResetError.

urllib wraps some connection failures in URLError, which the CLI already
handled in six places, and lets the rest through as bare OSError subclasses.
Two exposures: urlopen() itself, and the reads after a successful open --
the download loop and the debug tail loop both call resp.read() in a loop
with no handler at all, which is exactly where a reboot mid-transfer lands.

CONNECTION_DROPPED names the family once, including IncompleteRead, which a
chunked body cut short raises instead of an OSError. request_json re-raises
it as URLError so the six existing handlers keep working and each still
names its own operation; the two streaming loops report through _dropped().
A dropped download leaves the partial file in place, which is what --resume
is for.

DroppedConnectionTests adds a socket server that resets with SO_LINGER 0, so
the client sees a real ECONNRESET rather than a clean close, and covers a
reset before any reply, on a JSON command, mid-download and on health. Each
asserts exit 8, no traceback, and exactly one error: line. Run against the
unfixed CLI all four error with the traceback the story was filed for, so
they fail for the right reason. Suite: 119 tests, all passing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016emgLTZm7w3qZsRbukWwE4
After a timeout the ST re-sends the same 1 KB chunk with a fresh token, and
the RP treated it as new data and appended it again: the file gained a
duplicated chunk and lost its tail. Neither side carried anything that could
tell the two apart -- a retried chunk and a new one are identical on the
wire.

An absolute offset does not work without more state than it appears: the
m68k knows only how far it has got within the current Fwrite call, not the
file position, which the RP owns and Fseek moves. A relative offset cannot
distinguish a retry of a call's first chunk from the first chunk of the next
call, since both are zero.

So the m68k carries a number that never repeats. write_seq is a longword in
the GEMDRIVE blob, bumped once per chunk and never per retry, never reset,
sent in d5 -- which send_sync_write_command_to_sidecart always transmits and
the RP was skipping. Each open file slot on the RP remembers the last
sequence it accepted and the bytes it wrote; a chunk whose sequence matches
is answered with that same count and not written again. The memo is stored
before the answer is sent, because the answer is what goes missing.

Verified on hardware with a debug hook that stalls the answer after the data
is committed. Copying a 512 KB file with two chunks stalled 10 s each, the
ST re-sent both, the RP reported "repeat of chunk N, answering 1024 again"
for each, and the copy is byte-identical to the source. A first attempt at
2 s stalls proved nothing -- no retry happened at all, because that is
inside the ST's COMMAND_TIMEOUT, and the matching checksum would have been a
false pass.

The cartridge image is now 10,144 of 10,240 bytes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016emgLTZm7w3qZsRbukWwE4
@diegoparrilla
diegoparrilla merged commit 3385ac1 into release/v1.1.0 Sep 17, 2026
2 checks passed
diegoparrilla added a commit that referenced this pull request Sep 17, 2026
@diegoparrilla
diegoparrilla deleted the epic/15-sd-and-cli-rough-edges branch September 17, 2026 17:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant