CAS improvements - #2300
Open
filimonov wants to merge 24 commits into
Open
Conversation
…blication Two defects surfaced by the `content_addressed_garbage_collection_log` scenario cards for issue #2233. Any `S3_ERROR` timeout during a GC round was recorded as an indistinguishable `Failed` outcome with a free-text error, and a failed round zeroed out the real counters and cleared `i_am_leader`, suppressing the heartbeat and provoking leadership ping-pong on a flaky backend. Transient error codes (`S3_ERROR`, `NETWORK_ERROR`, `ABORTED`, timeouts, `MEMORY_LIMIT_EXCEEDED`) now produce an `Aborted` outcome while keeping leadership, and `system.cas_gc_log` gains an `error_code` column alongside the `Aborted` outcome. Separately, the emulated blob-publication path materialized the whole blob body in memory (about 1 GiB for a 512 MiB blob) under a global mutex; it now streams the body instead. Related: #2233 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
…_ERROR A fetch-by-relink that loses the offer-to-confirm race -- the source's ref moved (a merge, a mutation, an outdated-part drop) between the offer and the confirm -- is a designed, fail-closed outcome: the receiver abandons the relink and the replication queue retries, re-selecting the source and the covering part. It was thrown as `NETWORK_ERROR`, which misdescribes it three ways: - both queue executors (`processQueueEntry`, `ReplicatedMergeTreeQueue`-driven `ReplicatedMergeMutateTaskBase`) treat `NETWORK_ERROR` as an unclassified failure, so every refusal printed an Error-level log line with a full stack trace; issue #2219 records a multi-hour false triage chasing a network fault that was never there (up to 53% of relink proofs refuse under small-part load); - stateless `part_log` hygiene checks tolerate the fetch-transient class under the code upstream fetches use for it, `NO_REPLICA_HAS_PART` (e.g. `02265_column_ttl` whitelists exactly that code), so a refusal landing in `part_log` as `NETWORK_ERROR` fails them -- this is what broke `02265_column_ttl` in the CAS lanes on PR #2159 (13/14 reruns under `prefer_fetch_merged_part_size_threshold=1`); - the label suggests retrying the transport, while the one recovery that is unsound here is a byte re-request to the same source. Both relink retry-later throw sites (taxonomy row 3, the confirm refusal, and row 5b, the unresolved promote) now throw `NO_REPLICA_HAS_PART`. The queue behavior is unchanged -- the exception is stored on the entry, backed off, and re-executed -- but both executors demote it to INFO with no stack trace. Unlike `ABORTED` (the other demoted code), it keeps `need_to_save_exception`, so a refusal storm stays visible in `system.replication_queue`; `ABORTED`'s save-nothing shape is the known pathology where a refusal loop runs invisibly with no backoff accounting. `test_confirm_refuses_when_source_dropped_in_window` now pins the classification: the refusal must not appear at Error level, must appear at Information level, and must reach `part_log` only as `NO_REPLICA_HAS_PART`. No message text changed; no generic queue code changed. Closes: #2219 Related: #2159 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
…eys renamed yet) Behavior-preserving groundwork for an atomic rename of the CAS wire-format JSON keys from abstract letters (`t`, `k`, `s`, ...) to semantic names (`kind`, `outcome`, `state`, ...), landed as its own phase so the rename itself is a single reviewable diff. Adds `WireKey` and per-encoding field write helpers, and `EnumWireTable` — a table pairing each enum value with its wire word, proven complete against the enum by a set-equality coverage check with a failing witness for every member. `kMinBlobHeaderLen` gets one compile-time owner instead of several hand-kept constants. `TokenType`, `ObjectKind`, and `BlobHashAlgo` move onto `EnumWireTable`, and the blob-meta, pool-meta, GC state/heartbeat/ maintenance, server-root, blob-envelope, ref-log/ref-ckpt/ref-snapshot/ ref-catalog, run, fold-seal, and gc-outcomes codecs are all migrated onto the carriers — every one of them still writing its existing wire spelling. `RunMarker` becomes a typed enum, and the format test battery is closed out with a set-equality check over the codec registry. No wire-format bytes change in this phase; the follow-up phase (next commit) performs the actual key cut. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
The actual key rename, on top of the phase-1 carrier infrastructure. The
format generation history is first reset to a `{1, 1}` baseline, since CAS
has no released, persisted data yet and pre-release generations exist only
to prove the evolution machinery.
Every CAS wire format switches its JSON keys from single-letter/abbreviated
spellings to descriptive names in one pass: the shared `BlobRef`/`Token`/
`ManifestRef`/binding fields, `cas_blob_meta`, `cas_pool_meta` (`algos_used`
becomes a JSON word array instead of a bitmask), GC state/heartbeat/
maintenance state, the server-root record (`MountLease::min_active` becomes
`min_active_build_sequence`), `cas_ref_ckpt`, `cas_ref_log` (the seal link
becomes `!prev_epoch`/`!prev_seq`), `cas_ref_snapshot`, `cas_part_manifest`,
`cas_run`, `cas_gc_outcomes` (`kind`/`outcome`), the fold-seal record and its
`CoverageClass` words, the blob descriptor (with its 239-byte worst case
proved at compile time against the 240-byte floor), and `cas_ref_catalog`.
Golden tests are re-pinned to the new bytes throughout.
Token-group requiredness is unified through `TokenFields::build`: an outcome
missing its token now fails closed instead of serializing a partial group.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
…enchmark Small correctness and review follow-ups to the wire-key cut: the part manifest names its namespace field the way every other object does, the algorithm set is read from the proven `EnumWireTable` instead of two independent hand-kept lists, the GC lease and heartbeat keep their separate owner spellings (documented, not merged), and the wire-format word writer gets the contract it was always assumed to have. Extends the `benchmark_cas_ref_protocol` harness to cover every format and direction the wire-keys design measures, plus a review-round fix to that harness. Also fixes `c++expr`: the generated work function needs internal linkage, without which ClickHouse-mode compilation did not work. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
…speedup Full before/after measurement of the wire-key cut found decode of four of the five formats barely slower, with `cas_fold_seal` the exception (its short strings make the longer keys dominate). Chasing that, the JSON object reader and the per-format row reader are now reused across a stream's rows instead of rebuilt for each one, cutting decode time 57-81% (53-79% net of the key-length cost). A separate copy-free string-read attempt was measured at a 6-7% regression on `cas_ref_catalog` and is not included here. Also lets the full stateless test suite run locally: `functional_tests.py` turns on verbose output for the dataset-attach step (so a `DNS_ERROR` that only fires outside CI doesn't get swallowed and misread as a Kafka failure downstream) and extends the "skip stateful tests when running locally" guard to a local run with no test selector at all. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
`ALTER TABLE ... EXPORT PARTITION` was refused with `SUPPORT_IS_DISABLED` on a content-addressed disk because it is absent from the partition-command allowlist in `MergeTreeData`. The rejection it fell into says the command "clones parts file-by-file with no transaction, which would corrupt the clone", and that reason does not describe exporting. `ExportPartTask` reads the source part through `MergeTreeSequentialSource` (`MergeTreeSequentialSourceType::Export`) under `readLockParts` and writes rows into the destination through a `SinkToStorage` on an ordinary query pipeline. Nothing is hard-linked or copied on the source disk; the command's own bookkeeping is in ZooKeeper. So the allowlist was rejecting it by omission rather than by an argument that applies to it, which the code around it already half concedes: `EXPORT_PARTITION` is listed among the commands permitted to target `PARTITION ALL` a few lines above. Verified end to end rather than by inspection, on a server built from this change: a `ReplicatedMergeTree` source on a CAS disk holding (1,2020), (2,2020), (3,2021), exported to an `IcebergLocal` destination. `EXPORT PARTITION ID '2020'` succeeds and the destination holds exactly (1,2020) and (2,2020) — the right partition, and the 2021 row correctly absent. Two limitations surfaced on the way and are NOT addressed here, because neither is about CAS. Export is implemented only for `ReplicatedMergeTree`: a plain `MergeTree` source now returns `Code: 48 NOT_IMPLEMENTED` instead of the CAS refusal, so the reproduction in the report — which uses a plain MergeTree — will still fail, just for its real reason. And the operation remains behind the server setting `allow_experimental_export_merge_tree_partition`. Closes: #2291 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KiHKrvEVy8u4nA1A8qYFUY Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
… answer On a generation-dialect (GCS) mount, `ObjectStorageBackend::checkPoolPreconditions` refused to mount both when the bucket was verified to have versioning enabled and when the probe simply couldn't get an answer. The first live run against a real GCS bucket hit the second case: the service account lacked `storage.buckets.get`, `GetBucketVersioning` returned 403, and every writable CAS mount on that bucket failed with `NOT_IMPLEMENTED` at server start — a missing IAM grant turned into a hard outage, even though an unreadable bucket configuration is not evidence the bucket is actually versioned. The probe now logs a warning naming what it couldn't verify and how to fix it (grant `storage.buckets.get`, or confirm by hand) and lets the mount proceed. A bucket confirmed versioned still refuses, because a token-exact `DELETE` there archives a noncurrent generation instead of reclaiming storage. That first credentialed run against Google (HMAC groups) also surfaced three test-suite assumptions the real service doesn't meet (`system.cas_log.token` isn't always a numeric generation for build-lifecycle rows; a second `COUNT()` over a Parquet object is answered from the per-file row-count cache, not the Parquet metadata cache; a disk over an absent bucket refuses at `CREATE TABLE`, not the first `INSERT`) and one open question — whether process-wide `system.events` deltas can be attributed to one statement when a mount-lease renewal shares the same counters. The suite now attributes every counter it asserts through `system.query_log.ProfileEvents` instead. Also documents GCS's request-rate limits in the CAS bucket requirements. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
… whole namespace Same-pool replication transfers only a part's manifest: the receiver publishes its own ref over the sender's blobs, then asks the sender a read-only question — "do you still hold exactly this manifest for this part?" — and promotes only on `Yes`. Rule 3 of `CasRefLedger::confirmExactRef` answered `Unknown` whenever ANY mutation of the same namespace was queued, in flight, or awaiting its checkpoint frontier, not just a mutation of the asked-about ref. On the live GCS stand, two replicas answered each other `Unknown` almost every time for forty minutes: every replica is also a receiver, and each failed fetch appends two records to its own lane (a precommit, then its removal on abort), so under load neither side ever observed the other's lane quiet. Both replication queues wedged at 1.5-1.7k entries, the replicas diverged to 123k against 166k rows, and the soak died on `SYSTEM SYNC REPLICA`. Nothing was lost — once one side stopped fetching, the other drained in two minutes — but the lane-wide refusal made every sustained-write workload look like data loss in progress. On RustFS in a LAN the window closing this fast never showed the defect; GCS limits checkpoint publication to about one mutation per second per object, so the window is long enough to matter. Rule 3 now refuses only when the asked-about ref itself has a queued or in-flight mutation, via `RefTableRuntime::carved` mirroring the tenure's carved items and validating a ref-scoped item's ops against its `MutationScope` before durability. Covered by a two-node liveness case against a fake GCS with delayed `_ckpt` writes (`test_cas_gcs_relink_liveness`), and every confirm refusal is now attributed and counted rather than silent. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
A manifest id names one content, ever: `stageManifest` mints it from `(writer_epoch, build_sequence, next_manifest_ordinal++)` — all three either a durable counter bumped by a conditional write or strictly increasing per process — and writes the body exactly once. The only other mutations of a manifest are exact-token deletes (writer cleanup, GC's owner-removed cleanup, the orphan sweep). So the token carried in `ManifestCacheKey` distinguished nothing, and the `HEAD` that supplied it (`CasManifestReader::readManifestShared`) was a per-read check of a GC-side invariant, not of the cached content's validity — it cost one serial round trip per uncached or `ForceFresh` access and could only ever detect a protocol violation (something deleting a manifest the ref graph still names), never serve wrong bytes if removed, since id-to-content is a function. The cache now keys by `ManifestId` alone: no `HEAD` on a hit, exactly one `GET` on a miss. Detection of a dangling reference moves from "the next read" to "the first uncached read, or fsck". The `part_folder_validate` setting, which existed only to pace that now-removed `HEAD`, is retired along with it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Six review rounds across two earlier designs found nine classes of defect in how CAS
handled its conditional-write tokens, eight of them tracing to one root: `Token` was
`struct { String value; TokenType type; }`, anyone could construct one from anything, and
the backend accepted whatever it was handed. Concretely this let an empty value pass as a
token (and on S3/Azure an empty `If-Match` is *omitted*, so a fenced write silently becomes
an unconditional overwrite), let a write commit against a token that was really the result
of a later, unrelated `HEAD`, and let `TokenMismatch` — documented as remote evidence that
another incarnation is current — be returned for what was actually a local refusal, with GC
acting on it and mislabelling live blobs `Replaced`. Every earlier revision patched one
symptom at one call site; the next review round found the same root through a different
one. A second, independent waste rode along: `Backend::get` always issued both a `HEAD` and
a `GET`, though a `GET` already returns everything a `HEAD` does plus the body — doubling
the request cost of every control-object read.
This introduces the replacement, starting with its core (the migration of every CAS
subsystem onto it is the next commit): `Backend` becomes a string-in/string-out transport
callable only through a `TransportAccess` key; `Incarnation` replaces the free-form `Token`
as a type that can only be minted by the backend from an actual store response;
`CasRequests` owns a backend and a `Fence`, and `admit()`/`resume(generation)` hand out a
`CasOperation` carrying the admitted generation and an optional liveness predicate. Every
verb on that operation (`read`, `head`, `list`, `remove`, `publish`, `create`, `replace`,
`readModifyWrite`, ...) takes a `Retry` policy; the engine re-checks admission before every
attempt, before every sleep, and once more after a proven commit, settles every conflict and
ambiguity by one exact read, and reports one of `Committed | Declined | Conflict | Refused |
GaveUp` — never an exception for an ordinary lost race. An upstream slice under `src/IO`
and `S3ObjectStorage` adds a `SingleAttempt` request mode so a marked `GET` answers with the
same incarnation identity a `HEAD` does (closing the two-request cost) and a reissue that
gets back a different ETag is treated as body drift, not silently accepted. The old
controller stays in place during the migration; the next commits delete it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Follows the previous commit's engine introduction by moving every production caller off the old ad-hoc backend controller and onto `CasOperation`: pool bootstrap (sentinel probe, capability probe, plain objects, pool meta, manifest and ref-protocol readers), GC (maintenance state, namespace janitor, decommission, the GC core's lease/heartbeat/commits/ folds/persisted redelete), part-write (blob meta, the part-write transaction, create-first marker reconciliation), the ref lane (catalog and checkpoint publisher, namespace creation lifecycle, the resumed-operations arms, the catalog erase loop), and mount (renew, farewell, claim, epoch allocation, the heartbeat floor, remount re-anchoring). `PersistedIncarnation` replaces the ad-hoc token in the wire vocabulary, the record stream, the outcomes and the condemned rows. Each subsystem's move keeps its behavior but inherits the engine's guarantees for free: every write is admitted under a fence and re-checked before each attempt/sleep/commit, every conflict is settled by one exact read instead of an assumed outcome, and a credential refresh mid-attempt is never mistaken for a landed write. Along the way this fixes real bugs the engine surfaces mechanically rather than by inspection — e.g. two double-counting fault-injection doubles in the GC maintenance-state path, and several sites that treated an unobserved conflict as corruption instead of "vanished or a competing leader also wrote". The bulk of the diff is the matching migration of every test double (the `cp4` series) off the legacy backend overrides and onto the primitives the production code now actually calls — direct-Backend doubles for the primitives, virtualized clocks for every retry/backoff path that used to sleep for real, and fault injection that latches instead of pinning `max_attempts`, so a shut gate can no longer hang the test binary. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
… the engine, delete the old controller
Two closing passes over the request-engine migration. First, a naming decision: `Incarnation`
becomes `Etag` (`PersistedIncarnation` -> `PersistedEtag`, `CasIncarnation.{h,cpp}` ->
`CasEtag.{h,cpp}`) and `TokenType` folds into a `Dialect` alias, because the class was
carrying its wire field's name rather than its actual role — an ETag on S3-compatible
stores, a generation on GCS's JSON dialect, a minted sequence value on the emulated
backends. This does not touch the blob envelope's `incarnation_tag` or the catalog's
incarnation namespace, which are unrelated concepts the rename exists to stop colliding
with.
Second, the entire gtest suite (~120 files) moves off the legacy backend overrides and onto
`CasOperation`/`CasRequests`, naming the etag and the listed key the way production code now
does. With every test migrated, the old controller and its 1500-line test file
(`gtest_cas_request_control.cpp`) are deleted outright — this was the last thing keeping it
alive. `MountLeaseKeeper` is renamed to `MountLeaseRenewer` in the same pass (it renews a
lease; it is not ClickHouse's Keeper, and the old name kept reading as if it were).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
…gration Small correctness and hardening fixes found while the engine migration was under review: a read now ends only on an authoritative absence, not on every unretryable code; a raced `claimMount` reports the occupant it actually observed instead of its own proposal; an absent-key read through the request engine is no longer logged as an error (it's an ordinary outcome the engine already models); a hand-written retry loop now freezes one deadline and shares it across every call it makes, instead of re-deriving it per call; and an unobserved conflict is named for what it actually is — a vanish or a competing leader, never assumed corruption. The bulk of this is test hardening that follows from the engine actually enforcing pacing and admission where the old ad-hoc calls didn't: transport-fault doubles now inject `Poco::TimeoutException` (what production code actually throws) instead of `std::runtime_error`; several tests that asserted a schedule the engine never promised, or counted requests instead of asserting an outcome, are corrected; retry/backoff-dependent tests get their own virtual clock so they assert the engine actually reissued, rather than timing a real sleep; and the throttling coverage gate gains both a unit and an integration leg. Two properties orphaned by the old controller's test-file deletion (previous commit) are restored under the new API. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
…gc_read_concurrency) The fold's `fold_ref_intake` and `fold_reduce` phases issue their checkpoint, walk-position, manifest-edge and zero-in-degree-HEAD reads one at a time, in the round's own decision order — a live-GCS soak measured `fold_ref_intake` at 2303 s of a 4352 s phase wall (53%), and a separate finding recorded one fold round holding the GC lease for hours on a real bucket, still unfinished after 97 minutes. `GcReadAhead` sits in front of the fold's one admitted `CasOperation`: callers hint keys the sequential walk will need next, workers fetch them on a bounded pool under the same admitted generation, and the walk takes results at exactly the sites and in exactly the order it reads today — no decision, decode, counter or event moves off the round thread. A key nobody hinted is still read inline. Concurrency 1 issues no hints and is byte-for-byte today's behavior; the new `cas_gc_read_concurrency` setting is plumbed like `gc_meta_pool_size` and refused at 0 like `gc_shards`, with three `ProfileEvent`s for hits, misses and wasted results. Measured against a fixed per-request latency: `fold_ref_intake` 2.4x, `fold_reduce` 1.2x, the round overall 1.65x. Intake's speedup stops there because the round issues ref-log and manifest `GET`s one to one and a manifest key is only known once its log is decoded — that chain, and the graduation gate's inline meta re-check, are recorded as follow-up items. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Fetch-by-relink existed but was opportunistic: the receiver advertised ONE guessed pool identity before it knew where the sender actually kept the part (the caller's `dest_disk` if content-addressed, else the first content-addressed disk of the table's storage policy), then reserved the target disk the ordinary way — the TTL move rule's destination, `balancedReservation`, or the first volume with space — and accepted the relink offer only when the reservation happened to land on a disk of the advertised pool. Otherwise it re-requested the bytes. So a part already in the shared pool moved as bytes whenever the policy's placement disagreed with the guess: a tiered policy whose local volume comes first, a TTL rule naming the local tier for a fresh part, a policy holding two pools with the sender's in second place. The relink is the whole point of a shared pool — a fetch should move no bytes — and the storage policy could veto it by accident. The receiver now advertises every pool of its storage policy (any volume; a disk configured on the server but absent from the policy is not a candidate, since a part on it wouldn't load at startup), the sender names the one it matched, and the part lands on that pool's disk ahead of volume order, JBOD balancing and TTL move rules — the mover carries it to a TTL destination afterwards, the same way `perform_ttl_move_on_insert=0` already places first and moves later. A caller-supplied `dest_disk` (zero-copy `MOVE`) stays authoritative and untouched; a content-addressed disk never enters that path (`supportZeroCopyReplication()` is false for CAS). A read-only or broken disk on the right pool is not a candidate — nothing can publish a ref there. The offered pool must itself be an advertised pool (not matched by disk name), and the confirm's gate 0 compares mounts rather than disk names, closing a second-order gap the first pass left. Non-live pool disk = fail-close: a disk whose mount isn't live is left out of both the advertise and the placement, never guessed at. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
`ContentAddressedMetadataStorage::shutdown` contained two unbounded waits on the GC round: a `std::lock_guard` on the same mutex a synchronous round (`SYSTEM CAS GC`, `GC REBUILD`) holds for its whole duration, and `CasGcScheduler::stop`'s join, which the scheduler loop only even looks at at the top of its wait — a round already in flight never observes it, and a comment in the loop recorded an accepted extra full round if `stop` lands while the loop is blocked behind a manual round. A round has no wall-clock budget at all: `GcRoundWorkBudget` caps destructive work, not time, and against a slow bucket the wall clock is whatever the bucket makes it. Nothing in this wait protects durable state — the round is one-pass, committed by a single `gc/state` conditional write at the end, so an interrupted round is a crash the protocol already survives — the wait existed purely so no thread would touch a freed object. Shutdown and the storage destructor now arm the pool's teardown flag before the lock or join they would otherwise wait behind. The open request plane carries that flag as its fence, so a round in flight is refused at its next request, its next retry sleep, or its next streamed refill — the check lives at the request because a phase is long from making thousands of requests, not from making one long one, and `CasOperation` already re-checks admission before every attempt and sleep. Every join and every object's ownership stay unchanged: the join became short, not optional. A round cut this way is recorded `Stopped` rather than `Aborted`. Decommission is deliberately not armed: an already-latched self-remount completes one more step whose pool-identity probe runs on the open plane, and no arm point early enough to bound the GC join leaves that step intact. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
…bulk delete) A 15-minute real-GCS soak measured a sweep round costing 300-617 s per phase, all from per-object request loops on keys that are write-once by construction — an object whose only writer mints it once and whose only other mutations are exact-token deletes, so nothing about it needs re-reading once known. Three loops dominated: `fold_reduce`'s `GET` volume (2870-3400 per round) turned out to be the sweep's mount-floor probing, not manifest bodies — `floorForNamespace` reads the mount key of every `/`-prefix of a namespace for every listed manifest, though the floor is one value per server root; `manifest_deletes` cost 617 s for 3250 sequential conditional deletes at ~190 ms each; and `ref_object_cleanup` cost 199-204 s for 512-516 keys at four requests each. The fix cuts each loop to what the write-once property actually allows: one mount-floor read per namespace per sweep page (memoized), manifest bodies read only for nominated orphans and through the existing read-ahead instead of on every listed key, and a new write-once bulk-delete verb (`removeManyWriteOnce`, backed by `DeleteObjects` where the store has it) replacing the sequential per-key deletes for owner-removed manifests and for ref-object cleanup, which now revalidates its cohorts before batching them. None of this changes what gets deleted or when a namespace or manifest is judged eligible — only how many requests that judgment costs. Measured on real GCS: `fold_reduce` 300-380 s -> 2-5 s; `manifest_deletes` 617 s -> 2 s on a 1506-key round; `ref_object_cleanup` 204 s -> under 1 s. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
…own writes to a hot control object Every `CREATE TABLE`/`DROP TABLE` on a content-addressed disk mutates one pool-wide object, `cas/ref_catalog`, through a conditional write. Measured on ten parallel stateless jobs: `DROP TABLE` p50 2.4 s, p90 11.9 s, max 34.7 s; 113 `PreconditionFailed` in 80 s from 53 threads; the losing writer alone racking up 35 attempts with gaps growing to the 5 s cap; one `DROP` losing eight races in a row for 15.4 s. `CasRefCatalog::casUpdateImpl` starts every write with a `GET` and paces a lost race with `Retry::backoff`, a schedule shared with transport faults — a writer that has lost several races sleeps for seconds while a fresh one starts at zero, so the oldest loser is the least likely to win next. Worse, every writer in one process races every other writer in the *same* process: compare-and-swap is only needed against other servers, so every intra-process race is pure waste, each costing a `GET`, a refused `PUT`, a resolve `GET` and a sleep. `CasHotKeys` sits above the request engine as one FIFO ticket per pool and key: writers to the same hot object queue instead of racing, their conditional writes are combined into one physical attempt where safe (as-if-serial semantics, a `Conflict` cascade on a lost race so combined members see the answer a serial retry would have given them), and a last-known- object cache lets a lane holder skip the leading `GET` under one rule. Losing a race against *another server* still paces with a flat jitter, not the transport-fault backoff. The GC erase over `ref_catalog` (`deleteCompletedRemovingAtSnapshot`) becomes the lane's first caller, and the pool owns the lane. This is phase A only — combining, spacing, the clamp and moving the GC erase itself onto the lane in full are follow-on work; the design and its 34 review revisions are recorded separately. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
…ed locals Several gtest fixtures declared a test-only backend hook or fault clock as a local, captured other locals in it by reference, and then let the store (declared before the hook) call the hook again during its own teardown — after the captured locals had already gone out of scope. Under ASan this is a use-after-scope: the store's destructor writes a "farewell" record that can invoke a still-armed hook whose captured references are already dead. Fixed by declaring the test clock/hook before the store that keeps calling it (two transient-round tests, the straggler-epoch test), and by clearing the checkpoint-advance recovery test's backend hook before the locals it captures die. A separate scripted S3 client fix allocates its response body with `Aws::New`, matching how the SDK actually frees it, instead of a mismatched allocator. Also corrects two suites that had started asserting a schedule the engine never promised. Also: the stateless CAS lanes now run the GC scheduler every 20 s instead of every 5 s, matching the interval those tests actually need. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
`S3ObjectStorage::getSingleAttemptClient` (`SingleAttemptRetryStrategy`, `max_retries=0`) and the write path carrying `WriteSettings::object_storage_retry_profile == SingleAttempt` belong to the CAS control plane's conditional writes: a failed attempt there is not the final answer — `CasOperation::writeLoop` resolves the outcome by a read and reissues — yet two upstream sites logged it at Error as if it were terminal: `Client`'s network-error handler and the non-412 `S3Exception` site in `WriteBufferFromS3`. Both now log at Debug when the client carries `SingleAttemptRetryStrategy` or the write carries the `SingleAttempt` profile; an ordinary client configured with zero retries by a user setting (no outer loop resolving it) keeps logging at Error, since for that caller the failure really is final. The neighbouring 412 (`isPreconditionFailedError`) branch drops from Info to Debug for the same reason: a conditional write losing its precondition is the caller's expected answer, not an operator-facing event. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi
… gtests `SnapshotPublisherLatchedAcrossChunks` had two independent, reachable races. With `snapshot_log_count_threshold` at 0 (only reachable in this test), `precommitAdd`'s post-commit trigger dispatches a background publisher whose capture can still be in flight when `promote`, moments later, becomes lane leader and moves the lane to `Writing` — a lost race then backs the publisher off, and this pool's frozen `boot_ms_fn` never advances past that backoff deadline, poisoning every later dispatch on the namespace for the rest of the test. Fixed by driving `precommitAdd`/`promote` directly instead of through the shared `publishEmptyPart` helper, draining and explicitly publishing between the two commits. Separately, the carve hook gated the leader on the publisher reaching its blocked `PUT`, but under contention the dispatch's own scheduling delay could outlast that wait's bound, letting a leader released by timeout (not by the capture it meant to prove) start chunk 2 before the publisher captured — fixed by gating on the publisher's capture instead, which is causally prior to the `PUT`. Verified with 20 isolated `gtest_repeat` iterations and two full `CAS*` gates (2437/2437 each), reproduced only under CPU contention after isolated repeats alone did not reproduce it. Separately: a fatal `ASSERT_*` between launching an `AppendCaller`/`Caller` thread and its explicit `join()` left `TestBody` with the thread still joinable, and `std::thread::~thread()` on a joinable thread calls `std::terminate`, aborting the whole `unit_tests_dbms` binary and discarding every test scheduled after it. Both structs now join in their destructor if still joinable, so a failed assertion costs one test instead of the whole gate. Also: `CASDetachedWork` now stops and drains its detached publisher before the locals its hooks read go out of scope (ASan stack-use-after-return on `fake_boot` via `boot_ms_fn`), the same class of bug as the earlier test-hook lifetime fixes. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
filimonov
marked this pull request as ready for review
September 4, 2026 23:53
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes: #2219
Closes: #2291
Related: #2233
Related: #2159
What this is
Follow-up to #2159 (the initial
casmetadata-storage subsystem): correctness fixes, aperformance/reliability overhaul of the backend request layer, a wire-format key rename, and
three merged features (GC fold read-ahead, forced relink on fetch, GC teardown no longer
blocking on a round), plus a first phase of write-lane serialization for hot control objects
and a GC-cost cut for write-once keys. 22 commits, ~300 files, +38.6k/-19.4k restricted to CAS
paths. This is a curated, squashed reconstruction of the
cas-gc-rebuilddevelopment tree:internal design docs, plans and the soak-harness (
utils/ca-soak) are intentionally not partof this PR (dev-only tooling, no shipped behavior).
How the series is structured
S3_ERROR/NETWORK_ERROR/timeout during a GC round now yieldsAborted(keeping leadership) insteadof an indistinguishable
Failedthat zeroed real counters and droppedi_am_leader; theemulated blob-publication path streams instead of materializing ~1 GiB in memory. (Related CAS: replica HTTP dies on green-path soak after relink NETWORK_ERROR storm #2233)
NO_REPLICA_HAS_PARTinstead ofNETWORK_ERRORfor a relink-confirm refusal — adesigned fail-closed outcome, not a network fault; fixes false Error-level triage and
part_loghygiene checks. (Closes CAS: relink proof refusals are logged at Error with a stack trace despite being an expected outcome #2219)WireKey,EnumWireTablewithset-equality coverage proofs) for every CAS wire-format codec; no bytes change yet.
abbreviated/single-letter keys to descriptive names (
kind,outcome,state, ...).decode), plus fixes that let the full stateless suite run locally.
cas: recommend single-replica merges (doc).ALTER TABLE ... EXPORT PARTITIONnow works from a source on a CAS disk — it readsthrough a sequential source and writes via a sink, nothing is hardlinked on the source
disk, so the CAS refusal was by omission, not by substance. (Closes CAS: EXPORT PARTITION from a CAS source is rejected with SUPPORT_IS_DISABLED #2291)
storage.buckets.getIAM grant turned into a hard outage; now warns and continues. Plusthree live-GCS test-suite corrections found on the first credentialed run.
any mutation of the same namespace, which live-GCS testing showed livelocking two
replicas for 40 minutes under sustained write load; now scoped to the ref actually asked
about.
HEADon a cache hit (a manifest idnames its content, ever); retires the
part_folder_validatesetting that existed only topace that
HEAD.CasRequests/CasOperationrequest engine (core) — replaces the free-formToken(which let an empty value pass as a fenced condition, silently turning a conditional write
into an unconditional overwrite, among 8 other defect classes found across two design
reviews) with a type only the backend can mint, a deadline-bound retry engine, and a
SingleAttemptrequest mode that halves the request cost of a control-object read.matching test-double migration.
Incarnation→Etag/TokenType→Dialectrename, gtest suite onto the engine, deletethe old controller — naming the type by its actual role instead of its wire field;
~120 mechanical test-file migrations; retires the ~1500-line legacy controller.
cas_gc_read_concurrency) — overlaps the fold's small-objectround trips on a bounded pool without moving any decision off the round thread; measured
1.65x on the round overall (2.4x on
fold_ref_intake, which a live-GCS soak measuredtaking 97+ minutes unfinished).
every pool of its storage policy instead of one guess, so a shared-pool fetch never moves
bytes just because a TTL rule or volume order disagreed with the guess.
its next request/retry-sleep/refill via a teardown liveness carried on the open request
plane, recorded as
Stoppedrather thanAborted.instead of once per listed manifest, late/read-ahead manifest reads, and a bulk-delete verb
for owner-removed manifests and ref-object cleanup. Measured on real GCS:
fold_reduce300-380s → 2-5s,manifest_deletes617s → 2s,ref_object_cleanup204s → <1s.for
cas/ref_catalog's conditional writes, replacing same-process write races (measured:DROP TABLEp90 11.9s/max 34.7s, 113PreconditionFailedin 80s from 53 threads) withqueuing and combined commits.
teardown hook, plus a stateless-lane GC-scheduler interval tuning.
SingleAttemptconditional-write attempt is resolved by an outer retry loop, not terminal; logging it at
Error was a false-positive operator signal.
Verification
CAS*gtest gate green throughout (the request-engine migration alone: 2406/2406 at itsfinal checkpoint).
test_cas_replicated_relink,test_cas_gcs,test_cas_gc_sharded,test_cas_gc_bulk_deleteintegration suites.
bodies for exact figures); the hot-key write lane's contention measurements are from ten
parallel stateless CI jobs.
cas-gc-rebuildcommit each squash groupwas cut from is empty by construction (verified file-by-file before opening/updating this PR).
Developed with AI assistance (Claude); every commit carries
Co-Authored-ByandSigned-off-by.Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Improvements and fixes to the experimental content-addressed storage (
cas) metadata-storagetype:
ALTER TABLE ... EXPORT PARTITIONnow works from a CAS-disk source; fetch-by-relinkalways lands a same-pool part on that pool's disk instead of occasionally streaming its bytes;
a disk's teardown no longer blocks on an in-flight GC round; several GC-round cost cuts on
real object storage (round-trip counts down by 60-300x on the measured phases); a reliability
overhaul of the CAS-to-object-storage request layer closing several conditional-write edge
cases; and the CAS wire-format's internal JSON keys are renamed from abbreviations to
descriptive names (format generation reset; no compatibility concern, since CAS has no
released, persisted data yet).
Documentation entry for user-facing changes
Updated in this PR:
docs/en/antalya/cas/architecture/{backend,garbage-collection,manifests-and-refs,mounts-and-leases,read-path,replication,storage-layout}.md,docs/en/antalya/cas/{index,configuration,bucket-requirements}.md,docs/en/antalya/cas/operations/{debugging,monitoring,troubleshooting}.md,docs/en/operations/storing-data.md,docs/en/operations/system-tables/{cas_log,cas_gc_log}.md.CI/CD Options
Exclude tests:
Regression jobs to run: