[Nic-dorman]

Summary

antd's download streams could only return a whole object. A client that needed part of one, such as a Range request, a media seek or an S3 gateway serving a ranged GET, had to read from the start and throw bytes away. ant-core already reads byte ranges (Client::data_download_range fetches and decrypts only the chunks a range overlaps), so this PR exposes that through antd's REST and gRPC streams and adds the antd-go methods. Streaming and Range downloads in Indelible are blocked on this.

What changes

REST. POST /v1/data/stream (private, by DataMap) and GET /v1/data/public/{addr}/stream honour one Range: bytes= range, in any RFC 9110 form: a-b, a- or -n.

• Responses:
  • 206 with Content-Range.
  • 416 with Content-Range: bytes */size and code RANGE_NOT_SATISFIABLE when the start is at or past the end.
  • 400 for a malformed range, multiple ranges, or Range combined with the NDJSON progress format.
• Clamping: a last or -suffix past the end is clamped, as RFC 9110 says.
• Multiple ranges are rejected rather than answered with the whole object, which could be many GB the caller didn't ask for.
• Whole-object responses now send Accept-Ranges: bytes.

gRPC. Both stream requests gain optional uint64 offset = 3 and optional uint64 length = 4.

• Errors: a start past the end returns OUT_OF_RANGE. Length 0, or a range with include_progress, returns INVALID_ARGUMENT.
• Metadata: x-content-length (the range's length) and x-content-range.

Streaming with bounded buffers (antd/src/range.rs). data_download_range returns one buffer, so an open-ended range on a multi-GB object would otherwise buffer all of it. antd reads a range in windows of 8 chunks aligned to chunk boundaries, about 32 MiB of plaintext at most. It fetches the next window while the previous one drains. Every chunk is still fetched exactly once.

This bounds antd's own buffers per stream, not the process. ant-core keeps each chunk it fetches in its chunk cache, an LRU of 1,024 chunks that antd leaves at the default. A large read therefore grows antd's RSS by about 4 MiB per chunk, up to about 4 GiB. That is measured below. Capping the cache is a follow-up.

antd-go.

• DataStreamRange and DataStreamPublicRange on both the REST and gRPC clients. They return a *RangeReader (io.ReadCloser plus Offset, Length and Size), a stream rather than a buffer.
• RangeToEnd and *RangeNotSatisfiableError.
• A daemon that ignores the range is reported as an error, never silently read as the whole object.

Generated code and docs.

• OpenAPI documents Range, 206 and 416. It also adds the POST /v1/data/stream route, which was missing.
• Go stubs regenerated. This also picks up stale health and upload stubs that were already behind on main.
• Swift stubs regenerated with antd-swift/scripts/generate-protos.sh.
• Kotlin proto copy synced.
• antd-rust: generated code updated, plus ..Default::default() on 4 request literals so the new optional fields compile.

Compatibility

Additive. The proto fields are new and optional, and requests without a range behave exactly as before. No ant-core change is needed: the branch uses only public APIs of the pinned ant-core 0.10.0. It also compiles and passes unchanged against ant-core 0.11.0 and against WithAutonomi/ant-client#214's head, which rewrites the read engine underneath this call.

Test evidence

Unit tests: antd 111 pass (15 new: range parsing, clamping, window planning, error mapping), antd-rust 118 pass, antd-go all pass (10 new range tests). fmt, clippy -D warnings and doc are clean. After merging main, which brings the rustc 1.99 clippy fix from #312, antd and antd-rust tests, fmt and clippy were re-run locally, and all CI checks pass.

Multi-GB, on a 25-node devnet (at f105516): a 4.5 GiB object (4,831,850,553 bytes: 1,154 data chunks and a 3-chunk DataMap level), uploaded private and public. 38 ranges were read through antd-go over REST and gRPC, and every byte was compared with the source file.

• Coverage: ranges straddling 2^31 and 2^32, offsets past 2^32, chunk and 8-chunk window boundaries, the last partial chunk, EOF clamping, open-ended ranges, 12 random ranges up to 20 MiB, a 300 MiB range, a 1 GiB open-ended range from below 2^32, and the whole object as bytes=0-.

• Chunks fetched, counted from antd's chunk debug log for each range: every range fetched only the chunks it overlaps, each once. Overlapping chunks fetched by an earlier range came from ant-core's chunk cache. The first request also fetched the 3 DataMap chunks; later requests hit the cache.

  Range	Chunks overlapped	Fetched	From cache	Time
  1 KiB in chunk 300	1	1	0	114 ms
  32 B straddling 2^32	1	1	0	116 ms
  300 MiB in the middle (gRPC)	76	72	4	5.0 s
  1 GiB from 2^32 − 512 MiB to the end (gRPC)	258	246	12	16.2 s
  Whole object, bytes=0- (REST)	1,154	1,050	104	69 s (69.9 MB/s)

• Errors and raw HTTP forms: 416 / OUT_OF_RANGE at offset = size on both transports; bytes=-100 → 206 with the right Content-Range; an inclusive range across 2^32 → 206; bytes=size- → 416 with Content-Range: bytes */size; bytes=0-1,5-6 → 400. antd logged no ERROR or WARN lines.

• Memory: antd's RSS was about 520 MiB after the uploads. It peaked at 4,654 MiB during the whole-object read, which is that baseline plus a full 1,024-chunk cache. It grew by 3.4 to 4 MiB for each chunk fetched throughout the run. This is the chunk cache described above, not the range windows.

Earlier, on a 50 MiB object: 55 private and 9 public ranges over REST and gRPC, byte-identical, with curl confirming status codes and headers.

Through an S3 gateway prototype: a 64 MiB aws s3 cp download, which aws-cli splits into 8 parallel ranged parts:

Gateway read path	Cold (antd just restarted)	Warm
Whole stream, skip to offset	14.1 s	1.33 s
Range (this PR)	2.95 s	0.96 s

Follow-ups (not in this PR)

• ant-core bump. antd still pins ant-core 0.10.0. perf(read): share in-flight chunk fetches across concurrent reads ant-client#218 makes concurrent reads of one chunk share a single fetch. That removes the duplicate fetches parallel ranged GETs cause at part boundaries and on DataMap chunks. antd gets it from the ant-core bump, with no code change, and the bump is a separate PR.
• Chunk cache cap. antd uses ant-core's default 1,024-chunk cache, up to about 4 GiB, as measured above. A configurable, smaller cap would bound antd's memory on large reads.
• Core improvements: a streaming range API in ant-core, which would let antd drop its windowing, and native read-ahead for sequential range readers.

🤖 Generated with Claude Code

[dirvine]

Review — ant-sdk PR #313 (head f105516)

Verdict: APPROVE. Additive byte-range download support on antd's data streams (REST + gRPC + antd-go). Verified against a detached clone at the exact head; base b38c3d3 is an ancestor of head; all CI checks pass (antd, antd-rust, antd-go, antd-csharp build, antd-js, security audit, kotlin proto-copy, swift stub match).

What I verified

Core range logic (antd/src/range.rs). RFC 9110 parsing is correct for all three forms: a-b, a-, -n, with inclusive last clamped to size, suffix ranges clamped to the start, start >= size → 416, malformed/multi-range → 400 (including rejecting +-prefixed numbers and overflow). plan_windows is chunk-aligned: I traced the loop against range_records in ant-core 0.10.0 (ant-cli-v0.3.8) — both derive chunk offsets the same way (cumulative src_size, index-sorted), so windows partition the range exactly, each overlapping chunk is fetched exactly once, and the interior window boundaries fall on chunk boundaries. spawn_range_stream propagates a window error and ends the stream after delivered bytes — the same short-body contract as the whole-object path; dropping the receiver stops the producer.

REST (antd/src/rest/data.rs). 206 carries Content-Range: bytes first-last/size, Content-Length = range byte count, Accept-Ranges: bytes; 416 is generated by the RangeNotSatisfiable error mapping with Content-Range: bytes */size per RFC 9110 §15.5.17; malformed/multi-range and Range+NDJSON → 400; whole-object responses now advertise Accept-Ranges. Both routes resolve the child map to root before opening the stream, so sizes are the true plaintext sizes.

gRPC (antd/src/grpc/service.rs). offset/length are new optional fields (tags 3/4) on both stream requests — proto3 additive, old clients unaffected. Validation order is right (range+include_progress rejected before any fetch); x-content-length is the range byte count and x-content-range carries the object size; windows are re-framed into 1 MiB gRPC frames so 32 MiB windows don't exceed client receive limits; OUT_OF_RANGE/INVALID_ARGUMENT mapping matches the PR body and antd-go's expectations.

antd-go. RangeReader/DataStreamRange/DataStreamPublicRange on both REST and gRPC clients; 416 → *RangeNotSatisfiableError with size parsed from the 416's Content-Range (REST) or -1 (gRPC); a daemon that ignores the range (200 without Content-Range, or gRPC without x-content-range) is an error, never a silent whole-object read; client-side validation rejects negative offset / zero length before any request; rangeHeader guards the int64 overflow by degrading to bytes=offset-. Tests cover parsing, clamping, 416, ignored-range rejection, and the gRPC trailers-only vs legacy-daemon distinction.

Generated code. Go health.pb.go additions match health.proto tags 9–13 exactly (the stale-stub catch-up is faithful); the data.proto copy in antd-kotlin is byte-identical to antd's; Swift stubs regenerated (CI-verified); antd-rust generated code updated with ..Default::default() at the four request sites.

OpenAPI. Range parameter, 206/400/416 responses, RangeNotSatisfiable component, and the previously missing POST /v1/data/stream route are all documented and consistent with the handlers.

Non-blocking notes (none block merge)

1. antd-rust client surface — the new offset/length fields are generated but antd-rust's client doesn't yet expose ranged stream methods (antd-go is the first SDK client). Fine as scoped; a follow-up if antd-rust consumers want ranges.
2. gRPC 416 size not surfaced to Go callers — RangeNotSatisfiableError.Size is -1 on the gRPC path (the daemon sends the size in the status message text only). REST callers get the size from the 416 header. Cosmetic; a metadata trailer would be the fix if needed.
3. Chunk-cache memory — the ~4 GiB peak on whole-object reads is ant-core's default 1,024-chunk LRU, correctly disclosed in the PR body and tracked as a follow-up (configurable cap).
4. Stale Go health stubs — the regenerated health/upload stubs are faithful to the current protos, but folding a third change into this PR makes the diff noisier; future stub catch-ups could go in their own PR.

Follow-ups already tracked (not blockers)

ant-core bump (ant-client#218 dedupe), chunk-cache cap, streaming range API in ant-core.