Skip to content

CAS improvements - #2300

Open
filimonov wants to merge 24 commits into
antalya-26.6from
feature/antalya-26.6/CAS-improvements
Open

CAS improvements#2300
filimonov wants to merge 24 commits into
antalya-26.6from
feature/antalya-26.6/CAS-improvements

Conversation

@filimonov

@filimonov filimonov commented Sep 1, 2026

Copy link
Copy Markdown
Member

Closes: #2219
Closes: #2291
Related: #2233
Related: #2159

What this is

Follow-up to #2159 (the initial cas metadata-storage subsystem): correctness fixes, a
performance/reliability overhaul of the backend request layer, a wire-format key rename, and
three merged features (GC fold read-ahead, forced relink on fetch, GC teardown no longer
blocking on a round), plus a first phase of write-lane serialization for hot control objects
and a GC-cost cut for write-once keys. 22 commits, ~300 files, +38.6k/-19.4k restricted to CAS
paths. This is a curated, squashed reconstruction of the cas-gc-rebuild development tree:
internal design docs, plans and the soak-harness (utils/ca-soak) are intentionally not part
of this PR (dev-only tooling, no shipped behavior).

How the series is structured

  1. GC round transient-failure classification + streaming publish — an S3_ERROR/
    NETWORK_ERROR/timeout during a GC round now yields Aborted (keeping leadership) instead
    of an indistinguishable Failed that zeroed real counters and dropped i_am_leader; the
    emulated blob-publication path streams instead of materializing ~1 GiB in memory. (Related CAS: replica HTTP dies on green-path soak after relink NETWORK_ERROR storm #2233)
  2. NO_REPLICA_HAS_PART instead of NETWORK_ERROR for a relink-confirm refusal — a
    designed fail-closed outcome, not a network fault; fixes false Error-level triage and
    part_log hygiene checks. (Closes CAS: relink proof refusals are logged at Error with a stack trace despite being an expected outcome #2219)
  3. Wire-keys phase 1 — carrier infrastructure (WireKey, EnumWireTable with
    set-equality coverage proofs) for every CAS wire-format codec; no bytes change yet.
  4. Wire-keys phase 2 — the actual cut: every CAS JSON wire format moves from
    abbreviated/single-letter keys to descriptive names (kind, outcome, state, ...).
  5. Wire-keys phase 3 — review polish and the extended ref-protocol benchmark.
  6. Decode speedup — JSON object/row reader reuse across a stream's rows (57-81% faster
    decode), plus fixes that let the full stateless suite run locally.
  7. cas: recommend single-replica merges (doc).
  8. ALTER TABLE ... EXPORT PARTITION now works from a source on a CAS disk — it reads
    through a sequential source and writes via a sink, nothing is hardlinked on the source
    disk, so the CAS refusal was by omission, not by substance. (Closes CAS: EXPORT PARTITION from a CAS source is rejected with SUPPORT_IS_DISABLED #2291)
  9. GCS mount: don't refuse on an unanswerable versioning probe — a missing
    storage.buckets.get IAM grant turned into a hard outage; now warns and continues. Plus
    three live-GCS test-suite corrections found on the first credentialed run.
  10. Relink-confirm liveness scoped to the asked-about ref — the previous rule refused on
    any mutation of the same namespace, which live-GCS testing showed livelocking two
    replicas for 40 minutes under sustained write load; now scoped to the ref actually asked
    about.
  11. Manifest decode cache keyed by id alone — no HEAD on a cache hit (a manifest id
    names its content, ever); retires the part_folder_validate setting that existed only to
    pace that HEAD.
  12. CasRequests/CasOperation request engine (core) — replaces the free-form Token
    (which let an empty value pass as a fenced condition, silently turning a conditional write
    into an unconditional overwrite, among 8 other defect classes found across two design
    reviews) with a type only the backend can mint, a deadline-bound retry engine, and a
    SingleAttempt request mode that halves the request cost of a control-object read.
  13. Migrate every CAS subsystem onto the engine — pool, GC, write, ref, mount, plus the
    matching test-double migration.
  14. IncarnationEtag/TokenTypeDialect rename, gtest suite onto the engine, delete
    the old controller
    — naming the type by its actual role instead of its wire field;
    ~120 mechanical test-file migrations; retires the ~1500-line legacy controller.
  15. Post-engine review follow-ups and test hardening.
  16. GC fold read-ahead (cas_gc_read_concurrency) — overlaps the fold's small-object
    round trips on a bounded pool without moving any decision off the round thread; measured
    1.65x on the round overall (2.4x on fold_ref_intake, which a live-GCS soak measured
    taking 97+ minutes unfinished).
  17. A same-pool fetch always relinks onto the pool's disk — the receiver now advertises
    every pool of its storage policy instead of one guess, so a shared-pool fetch never moves
    bytes just because a TTL rule or volume order disagreed with the guess.
  18. A disk's teardown no longer waits out a GC round — an in-flight round is refused at
    its next request/retry-sleep/refill via a teardown liveness carried on the open request
    plane, recorded as Stopped rather than Aborted.
  19. GC round cost on write-once keys — a mount-floor read memoized per namespace per page
    instead of once per listed manifest, late/read-ahead manifest reads, and a bulk-delete verb
    for owner-removed manifests and ref-object cleanup. Measured on real GCS:
    fold_reduce 300-380s → 2-5s, manifest_deletes 617s → 2s, ref_object_cleanup 204s → <1s.
  20. Hot-key write lane, phase A — one FIFO ticket per pool+key above the request engine
    for cas/ref_catalog's conditional writes, replacing same-process write races (measured:
    DROP TABLE p90 11.9s/max 34.7s, 113 PreconditionFailed in 80s from 53 threads) with
    queuing and combined commits.
  21. ASan use-after-scope fixes in test fixtures whose captured locals outlived a store's
    teardown hook, plus a stateless-lane GC-scheduler interval tuning.
  22. Single-attempt S3 client logs its failed attempt below Error — a SingleAttempt
    conditional-write attempt is resolved by an outer retry loop, not terminal; logging it at
    Error was a false-positive operator signal.

Verification

  • CAS* gtest gate green throughout (the request-engine migration alone: 2406/2406 at its
    final checkpoint).
  • test_cas_replicated_relink, test_cas_gcs, test_cas_gc_sharded, test_cas_gc_bulk_delete
    integration suites.
  • Forced-relink and write-once-keys measurements taken on a real GCS soak stand (see commit
    bodies for exact figures); the hot-key write lane's contention measurements are from ten
    parallel stateless CI jobs.
  • The branch's CAS-restricted diff against the exact cas-gc-rebuild commit each squash group
    was cut from is empty by construction (verified file-by-file before opening/updating this PR).

Developed with AI assistance (Claude); every commit carries Co-Authored-By and
Signed-off-by.

Changelog category (leave one):

  • Experimental Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Improvements and fixes to the experimental content-addressed storage (cas) metadata-storage
type: ALTER TABLE ... EXPORT PARTITION now works from a CAS-disk source; fetch-by-relink
always lands a same-pool part on that pool's disk instead of occasionally streaming its bytes;
a disk's teardown no longer blocks on an in-flight GC round; several GC-round cost cuts on
real object storage (round-trip counts down by 60-300x on the measured phases); a reliability
overhaul of the CAS-to-object-storage request layer closing several conditional-write edge
cases; and the CAS wire-format's internal JSON keys are renamed from abbreviations to
descriptive names (format generation reset; no compatibility concern, since CAS has no
released, persisted data yet).

Documentation entry for user-facing changes

Updated in this PR: docs/en/antalya/cas/architecture/{backend,garbage-collection,manifests-and-refs,mounts-and-leases,read-path,replication,storage-layout}.md, docs/en/antalya/cas/{index,configuration,bucket-requirements}.md, docs/en/antalya/cas/operations/{debugging,monitoring,troubleshooting}.md, docs/en/operations/storing-data.md, docs/en/operations/system-tables/{cas_log,cas_gc_log}.md.

CI/CD Options

Exclude tests:

  • Fast test
  • Integration Tests
  • Stateless tests
  • Stateful tests
  • Unit tests
  • Performance tests
  • Aarch64 tests
  • All with ASAN
  • All with TSAN
  • All with MSAN
  • All with UBSAN
  • All with Coverage
  • All Regression
  • Disable CI Cache

Regression jobs to run:

  • Fast suites (mostly <1h)
  • Aggregate Functions (2h)
  • Alter (1.5h)
  • Benchmark (30m)
  • CAS (content-addressed storage; Antalya only)
  • ClickHouse Keeper (1h)
  • Iceberg (2h)
  • LDAP (1h)
  • OAuth (5m)
  • Parquet (1.5h)
  • RBAC (1.5h)
  • SSL Server (1h)
  • S3 (2h)
  • S3 Export (2h)
  • Swarms (30m)
  • Tiered Storage (2h)

filimonov and others added 8 commits September 2, 2026 01:01
…blication

Two defects surfaced by the `content_addressed_garbage_collection_log`
scenario cards for issue #2233. Any `S3_ERROR` timeout during a GC round was
recorded as an indistinguishable `Failed` outcome with a free-text error, and
a failed round zeroed out the real counters and cleared `i_am_leader`,
suppressing the heartbeat and provoking leadership ping-pong on a flaky
backend. Transient error codes (`S3_ERROR`, `NETWORK_ERROR`, `ABORTED`,
timeouts, `MEMORY_LIMIT_EXCEEDED`) now produce an `Aborted` outcome while
keeping leadership, and `system.cas_gc_log` gains an `error_code` column
alongside the `Aborted` outcome.

Separately, the emulated blob-publication path materialized the whole blob
body in memory (about 1 GiB for a 512 MiB blob) under a global mutex; it now
streams the body instead.

Related: #2233

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
…_ERROR

A fetch-by-relink that loses the offer-to-confirm race -- the source's ref
moved (a merge, a mutation, an outdated-part drop) between the offer and
the confirm -- is a designed, fail-closed outcome: the receiver abandons
the relink and the replication queue retries, re-selecting the source and
the covering part. It was thrown as `NETWORK_ERROR`, which misdescribes it
three ways:

- both queue executors (`processQueueEntry`,
  `ReplicatedMergeTreeQueue`-driven `ReplicatedMergeMutateTaskBase`) treat
  `NETWORK_ERROR` as an unclassified failure, so every refusal printed an
  Error-level log line with a full stack trace; issue #2219 records a
  multi-hour false triage chasing a network fault that was never there
  (up to 53% of relink proofs refuse under small-part load);
- stateless `part_log` hygiene checks tolerate the fetch-transient class
  under the code upstream fetches use for it, `NO_REPLICA_HAS_PART`
  (e.g. `02265_column_ttl` whitelists exactly that code), so a refusal
  landing in `part_log` as `NETWORK_ERROR` fails them -- this is what
  broke `02265_column_ttl` in the CAS lanes on PR #2159 (13/14 reruns
  under `prefer_fetch_merged_part_size_threshold=1`);
- the label suggests retrying the transport, while the one recovery that
  is unsound here is a byte re-request to the same source.

Both relink retry-later throw sites (taxonomy row 3, the confirm refusal,
and row 5b, the unresolved promote) now throw `NO_REPLICA_HAS_PART`. The
queue behavior is unchanged -- the exception is stored on the entry,
backed off, and re-executed -- but both executors demote it to INFO with
no stack trace. Unlike `ABORTED` (the other demoted code), it keeps
`need_to_save_exception`, so a refusal storm stays visible in
`system.replication_queue`; `ABORTED`'s save-nothing shape is the known
pathology where a refusal loop runs invisibly with no backoff accounting.

`test_confirm_refuses_when_source_dropped_in_window` now pins the
classification: the refusal must not appear at Error level, must appear at
Information level, and must reach `part_log` only as
`NO_REPLICA_HAS_PART`. No message text changed; no generic queue code
changed.

Closes: #2219
Related: #2159

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
…eys renamed yet)

Behavior-preserving groundwork for an atomic rename of the CAS wire-format
JSON keys from abstract letters (`t`, `k`, `s`, ...) to semantic names
(`kind`, `outcome`, `state`, ...), landed as its own phase so the rename
itself is a single reviewable diff.

Adds `WireKey` and per-encoding field write helpers, and `EnumWireTable` —
a table pairing each enum value with its wire word, proven complete against
the enum by a set-equality coverage check with a failing witness for every
member. `kMinBlobHeaderLen` gets one compile-time owner instead of several
hand-kept constants. `TokenType`, `ObjectKind`, and `BlobHashAlgo` move onto
`EnumWireTable`, and the blob-meta, pool-meta, GC state/heartbeat/
maintenance, server-root, blob-envelope, ref-log/ref-ckpt/ref-snapshot/
ref-catalog, run, fold-seal, and gc-outcomes codecs are all migrated onto the
carriers — every one of them still writing its existing wire spelling.
`RunMarker` becomes a typed enum, and the format test battery is closed out
with a set-equality check over the codec registry.

No wire-format bytes change in this phase; the follow-up phase (next commit)
performs the actual key cut.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
The actual key rename, on top of the phase-1 carrier infrastructure. The
format generation history is first reset to a `{1, 1}` baseline, since CAS
has no released, persisted data yet and pre-release generations exist only
to prove the evolution machinery.

Every CAS wire format switches its JSON keys from single-letter/abbreviated
spellings to descriptive names in one pass: the shared `BlobRef`/`Token`/
`ManifestRef`/binding fields, `cas_blob_meta`, `cas_pool_meta` (`algos_used`
becomes a JSON word array instead of a bitmask), GC state/heartbeat/
maintenance state, the server-root record (`MountLease::min_active` becomes
`min_active_build_sequence`), `cas_ref_ckpt`, `cas_ref_log` (the seal link
becomes `!prev_epoch`/`!prev_seq`), `cas_ref_snapshot`, `cas_part_manifest`,
`cas_run`, `cas_gc_outcomes` (`kind`/`outcome`), the fold-seal record and its
`CoverageClass` words, the blob descriptor (with its 239-byte worst case
proved at compile time against the 240-byte floor), and `cas_ref_catalog`.
Golden tests are re-pinned to the new bytes throughout.

Token-group requiredness is unified through `TokenFields::build`: an outcome
missing its token now fails closed instead of serializing a partial group.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
…enchmark

Small correctness and review follow-ups to the wire-key cut: the part
manifest names its namespace field the way every other object does, the
algorithm set is read from the proven `EnumWireTable` instead of two
independent hand-kept lists, the GC lease and heartbeat keep their separate
owner spellings (documented, not merged), and the wire-format word writer
gets the contract it was always assumed to have.

Extends the `benchmark_cas_ref_protocol` harness to cover every format and
direction the wire-keys design measures, plus a review-round fix to that
harness.

Also fixes `c++expr`: the generated work function needs internal linkage,
without which ClickHouse-mode compilation did not work.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
…speedup

Full before/after measurement of the wire-key cut found decode of four of
the five formats barely slower, with `cas_fold_seal` the exception (its
short strings make the longer keys dominate). Chasing that, the JSON object
reader and the per-format row reader are now reused across a stream's rows
instead of rebuilt for each one, cutting decode time 57-81% (53-79% net of
the key-length cost). A separate copy-free string-read attempt was measured
at a 6-7% regression on `cas_ref_catalog` and is not included here.

Also lets the full stateless test suite run locally: `functional_tests.py`
turns on verbose output for the dataset-attach step (so a `DNS_ERROR` that
only fires outside CI doesn't get swallowed and misread as a Kafka failure
downstream) and extends the "skip stateful tests when running locally" guard
to a local run with no test selector at all.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
`ALTER TABLE ... EXPORT PARTITION` was refused with `SUPPORT_IS_DISABLED` on a content-addressed disk
because it is absent from the partition-command allowlist in `MergeTreeData`. The rejection it fell
into says the command "clones parts file-by-file with no transaction, which would corrupt the clone",
and that reason does not describe exporting.

`ExportPartTask` reads the source part through `MergeTreeSequentialSource`
(`MergeTreeSequentialSourceType::Export`) under `readLockParts` and writes rows into the destination
through a `SinkToStorage` on an ordinary query pipeline. Nothing is hard-linked or copied on the
source disk; the command's own bookkeeping is in ZooKeeper. So the allowlist was rejecting it by
omission rather than by an argument that applies to it, which the code around it already half
concedes: `EXPORT_PARTITION` is listed among the commands permitted to target `PARTITION ALL` a few
lines above.

Verified end to end rather than by inspection, on a server built from this change: a
`ReplicatedMergeTree` source on a CAS disk holding (1,2020), (2,2020), (3,2021), exported to an
`IcebergLocal` destination. `EXPORT PARTITION ID '2020'` succeeds and the destination holds exactly
(1,2020) and (2,2020) — the right partition, and the 2021 row correctly absent.

Two limitations surfaced on the way and are NOT addressed here, because neither is about CAS. Export
is implemented only for `ReplicatedMergeTree`: a plain `MergeTree` source now returns `Code: 48
NOT_IMPLEMENTED` instead of the CAS refusal, so the reproduction in the report — which uses a plain
MergeTree — will still fail, just for its real reason. And the operation remains behind the server
setting `allow_experimental_export_merge_tree_partition`.

Closes: #2291

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KiHKrvEVy8u4nA1A8qYFUY
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown

Workflow [PR], commit [67d337f]

filimonov and others added 16 commits September 5, 2026 01:01
… answer

On a generation-dialect (GCS) mount, `ObjectStorageBackend::checkPoolPreconditions`
refused to mount both when the bucket was verified to have versioning enabled and when
the probe simply couldn't get an answer. The first live run against a real GCS bucket
hit the second case: the service account lacked `storage.buckets.get`, `GetBucketVersioning`
returned 403, and every writable CAS mount on that bucket failed with `NOT_IMPLEMENTED` at
server start — a missing IAM grant turned into a hard outage, even though an unreadable
bucket configuration is not evidence the bucket is actually versioned.

The probe now logs a warning naming what it couldn't verify and how to fix it (grant
`storage.buckets.get`, or confirm by hand) and lets the mount proceed. A bucket confirmed
versioned still refuses, because a token-exact `DELETE` there archives a noncurrent
generation instead of reclaiming storage.

That first credentialed run against Google (HMAC groups) also surfaced three test-suite
assumptions the real service doesn't meet (`system.cas_log.token` isn't always a numeric
generation for build-lifecycle rows; a second `COUNT()` over a Parquet object is answered
from the per-file row-count cache, not the Parquet metadata cache; a disk over an absent
bucket refuses at `CREATE TABLE`, not the first `INSERT`) and one open question — whether
process-wide `system.events` deltas can be attributed to one statement when a mount-lease
renewal shares the same counters. The suite now attributes every counter it asserts through
`system.query_log.ProfileEvents` instead. Also documents GCS's request-rate limits in the
CAS bucket requirements.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
… whole namespace

Same-pool replication transfers only a part's manifest: the receiver publishes its own
ref over the sender's blobs, then asks the sender a read-only question — "do you still
hold exactly this manifest for this part?" — and promotes only on `Yes`. Rule 3 of
`CasRefLedger::confirmExactRef` answered `Unknown` whenever ANY mutation of the same
namespace was queued, in flight, or awaiting its checkpoint frontier, not just a mutation
of the asked-about ref.

On the live GCS stand, two replicas answered each other `Unknown` almost every time for
forty minutes: every replica is also a receiver, and each failed fetch appends two records
to its own lane (a precommit, then its removal on abort), so under load neither side ever
observed the other's lane quiet. Both replication queues wedged at 1.5-1.7k entries, the
replicas diverged to 123k against 166k rows, and the soak died on `SYSTEM SYNC REPLICA`.
Nothing was lost — once one side stopped fetching, the other drained in two minutes — but
the lane-wide refusal made every sustained-write workload look like data loss in progress.
On RustFS in a LAN the window closing this fast never showed the defect; GCS limits
checkpoint publication to about one mutation per second per object, so the window is long
enough to matter.

Rule 3 now refuses only when the asked-about ref itself has a queued or in-flight mutation,
via `RefTableRuntime::carved` mirroring the tenure's carved items and validating a ref-scoped
item's ops against its `MutationScope` before durability. Covered by a two-node liveness case
against a fake GCS with delayed `_ckpt` writes (`test_cas_gcs_relink_liveness`), and every
confirm refusal is now attributed and counted rather than silent.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
A manifest id names one content, ever: `stageManifest` mints it from
`(writer_epoch, build_sequence, next_manifest_ordinal++)` — all three either a durable
counter bumped by a conditional write or strictly increasing per process — and writes the
body exactly once. The only other mutations of a manifest are exact-token deletes (writer
cleanup, GC's owner-removed cleanup, the orphan sweep). So the token carried in
`ManifestCacheKey` distinguished nothing, and the `HEAD` that supplied it
(`CasManifestReader::readManifestShared`) was a per-read check of a GC-side invariant, not
of the cached content's validity — it cost one serial round trip per uncached or
`ForceFresh` access and could only ever detect a protocol violation (something deleting a
manifest the ref graph still names), never serve wrong bytes if removed, since id-to-content
is a function.

The cache now keys by `ManifestId` alone: no `HEAD` on a hit, exactly one `GET` on a miss.
Detection of a dangling reference moves from "the next read" to "the first uncached read,
or fsck". The `part_folder_validate` setting, which existed only to pace that now-removed
`HEAD`, is retired along with it.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Six review rounds across two earlier designs found nine classes of defect in how CAS
handled its conditional-write tokens, eight of them tracing to one root: `Token` was
`struct { String value; TokenType type; }`, anyone could construct one from anything, and
the backend accepted whatever it was handed. Concretely this let an empty value pass as a
token (and on S3/Azure an empty `If-Match` is *omitted*, so a fenced write silently becomes
an unconditional overwrite), let a write commit against a token that was really the result
of a later, unrelated `HEAD`, and let `TokenMismatch` — documented as remote evidence that
another incarnation is current — be returned for what was actually a local refusal, with GC
acting on it and mislabelling live blobs `Replaced`. Every earlier revision patched one
symptom at one call site; the next review round found the same root through a different
one. A second, independent waste rode along: `Backend::get` always issued both a `HEAD` and
a `GET`, though a `GET` already returns everything a `HEAD` does plus the body — doubling
the request cost of every control-object read.

This introduces the replacement, starting with its core (the migration of every CAS
subsystem onto it is the next commit): `Backend` becomes a string-in/string-out transport
callable only through a `TransportAccess` key; `Incarnation` replaces the free-form `Token`
as a type that can only be minted by the backend from an actual store response;
`CasRequests` owns a backend and a `Fence`, and `admit()`/`resume(generation)` hand out a
`CasOperation` carrying the admitted generation and an optional liveness predicate. Every
verb on that operation (`read`, `head`, `list`, `remove`, `publish`, `create`, `replace`,
`readModifyWrite`, ...) takes a `Retry` policy; the engine re-checks admission before every
attempt, before every sleep, and once more after a proven commit, settles every conflict and
ambiguity by one exact read, and reports one of `Committed | Declined | Conflict | Refused |
GaveUp` — never an exception for an ordinary lost race. An upstream slice under `src/IO`
and `S3ObjectStorage` adds a `SingleAttempt` request mode so a marked `GET` answers with the
same incarnation identity a `HEAD` does (closing the two-request cost) and a reissue that
gets back a different ETag is treated as body drift, not silently accepted. The old
controller stays in place during the migration; the next commits delete it.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Follows the previous commit's engine introduction by moving every production caller off
the old ad-hoc backend controller and onto `CasOperation`: pool bootstrap (sentinel probe,
capability probe, plain objects, pool meta, manifest and ref-protocol readers), GC
(maintenance state, namespace janitor, decommission, the GC core's lease/heartbeat/commits/
folds/persisted redelete), part-write (blob meta, the part-write transaction, create-first
marker reconciliation), the ref lane (catalog and checkpoint publisher, namespace creation
lifecycle, the resumed-operations arms, the catalog erase loop), and mount (renew, farewell,
claim, epoch allocation, the heartbeat floor, remount re-anchoring). `PersistedIncarnation`
replaces the ad-hoc token in the wire vocabulary, the record stream, the outcomes and the
condemned rows.

Each subsystem's move keeps its behavior but inherits the engine's guarantees for free:
every write is admitted under a fence and re-checked before each attempt/sleep/commit, every
conflict is settled by one exact read instead of an assumed outcome, and a credential
refresh mid-attempt is never mistaken for a landed write. Along the way this fixes real
bugs the engine surfaces mechanically rather than by inspection — e.g. two double-counting
fault-injection doubles in the GC maintenance-state path, and several sites that treated an
unobserved conflict as corruption instead of "vanished or a competing leader also wrote".

The bulk of the diff is the matching migration of every test double (the `cp4` series) off
the legacy backend overrides and onto the primitives the production code now actually calls
— direct-Backend doubles for the primitives, virtualized clocks for every retry/backoff path
that used to sleep for real, and fault injection that latches instead of pinning
`max_attempts`, so a shut gate can no longer hang the test binary.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
… the engine, delete the old controller

Two closing passes over the request-engine migration. First, a naming decision: `Incarnation`
becomes `Etag` (`PersistedIncarnation` -> `PersistedEtag`, `CasIncarnation.{h,cpp}` ->
`CasEtag.{h,cpp}`) and `TokenType` folds into a `Dialect` alias, because the class was
carrying its wire field's name rather than its actual role — an ETag on S3-compatible
stores, a generation on GCS's JSON dialect, a minted sequence value on the emulated
backends. This does not touch the blob envelope's `incarnation_tag` or the catalog's
incarnation namespace, which are unrelated concepts the rename exists to stop colliding
with.

Second, the entire gtest suite (~120 files) moves off the legacy backend overrides and onto
`CasOperation`/`CasRequests`, naming the etag and the listed key the way production code now
does. With every test migrated, the old controller and its 1500-line test file
(`gtest_cas_request_control.cpp`) are deleted outright — this was the last thing keeping it
alive. `MountLeaseKeeper` is renamed to `MountLeaseRenewer` in the same pass (it renews a
lease; it is not ClickHouse's Keeper, and the old name kept reading as if it were).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
…gration

Small correctness and hardening fixes found while the engine migration was under review:
a read now ends only on an authoritative absence, not on every unretryable code; a raced
`claimMount` reports the occupant it actually observed instead of its own proposal; an
absent-key read through the request engine is no longer logged as an error (it's an
ordinary outcome the engine already models); a hand-written retry loop now freezes one
deadline and shares it across every call it makes, instead of re-deriving it per call; and
an unobserved conflict is named for what it actually is — a vanish or a competing leader,
never assumed corruption.

The bulk of this is test hardening that follows from the engine actually enforcing pacing
and admission where the old ad-hoc calls didn't: transport-fault doubles now inject
`Poco::TimeoutException` (what production code actually throws) instead of
`std::runtime_error`; several tests that asserted a schedule the engine never promised, or
counted requests instead of asserting an outcome, are corrected; retry/backoff-dependent
tests get their own virtual clock so they assert the engine actually reissued, rather than
timing a real sleep; and the throttling coverage gate gains both a unit and an integration
leg. Two properties orphaned by the old controller's test-file deletion (previous commit)
are restored under the new API.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
…gc_read_concurrency)

The fold's `fold_ref_intake` and `fold_reduce` phases issue their checkpoint, walk-position,
manifest-edge and zero-in-degree-HEAD reads one at a time, in the round's own decision
order — a live-GCS soak measured `fold_ref_intake` at 2303 s of a 4352 s phase wall (53%),
and a separate finding recorded one fold round holding the GC lease for hours on a real
bucket, still unfinished after 97 minutes.

`GcReadAhead` sits in front of the fold's one admitted `CasOperation`: callers hint keys the
sequential walk will need next, workers fetch them on a bounded pool under the same admitted
generation, and the walk takes results at exactly the sites and in exactly the order it
reads today — no decision, decode, counter or event moves off the round thread. A key
nobody hinted is still read inline. Concurrency 1 issues no hints and is byte-for-byte
today's behavior; the new `cas_gc_read_concurrency` setting is plumbed like
`gc_meta_pool_size` and refused at 0 like `gc_shards`, with three `ProfileEvent`s for hits,
misses and wasted results.

Measured against a fixed per-request latency: `fold_ref_intake` 2.4x, `fold_reduce` 1.2x, the
round overall 1.65x. Intake's speedup stops there because the round issues ref-log and
manifest `GET`s one to one and a manifest key is only known once its log is decoded — that
chain, and the graduation gate's inline meta re-check, are recorded as follow-up items.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
Fetch-by-relink existed but was opportunistic: the receiver advertised ONE guessed pool
identity before it knew where the sender actually kept the part (the caller's `dest_disk`
if content-addressed, else the first content-addressed disk of the table's storage policy),
then reserved the target disk the ordinary way — the TTL move rule's destination,
`balancedReservation`, or the first volume with space — and accepted the relink offer only
when the reservation happened to land on a disk of the advertised pool. Otherwise it
re-requested the bytes. So a part already in the shared pool moved as bytes whenever the
policy's placement disagreed with the guess: a tiered policy whose local volume comes
first, a TTL rule naming the local tier for a fresh part, a policy holding two pools with
the sender's in second place. The relink is the whole point of a shared pool — a fetch
should move no bytes — and the storage policy could veto it by accident.

The receiver now advertises every pool of its storage policy (any volume; a disk configured
on the server but absent from the policy is not a candidate, since a part on it wouldn't
load at startup), the sender names the one it matched, and the part lands on that pool's
disk ahead of volume order, JBOD balancing and TTL move rules — the mover carries it to a
TTL destination afterwards, the same way `perform_ttl_move_on_insert=0` already places first
and moves later. A caller-supplied `dest_disk` (zero-copy `MOVE`) stays authoritative and
untouched; a content-addressed disk never enters that path (`supportZeroCopyReplication()`
is false for CAS). A read-only or broken disk on the right pool is not a candidate — nothing
can publish a ref there.

The offered pool must itself be an advertised pool (not matched by disk name), and the
confirm's gate 0 compares mounts rather than disk names, closing a second-order gap the
first pass left.

Non-live pool disk = fail-close: a disk whose mount isn't live is left out of both the
advertise and the placement, never guessed at.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
`ContentAddressedMetadataStorage::shutdown` contained two unbounded waits on the GC round:
a `std::lock_guard` on the same mutex a synchronous round (`SYSTEM CAS GC`, `GC REBUILD`)
holds for its whole duration, and `CasGcScheduler::stop`'s join, which the scheduler loop
only even looks at at the top of its wait — a round already in flight never observes it,
and a comment in the loop recorded an accepted extra full round if `stop` lands while the
loop is blocked behind a manual round. A round has no wall-clock budget at all:
`GcRoundWorkBudget` caps destructive work, not time, and against a slow bucket the wall
clock is whatever the bucket makes it. Nothing in this wait protects durable state — the
round is one-pass, committed by a single `gc/state` conditional write at the end, so an
interrupted round is a crash the protocol already survives — the wait existed purely so no
thread would touch a freed object.

Shutdown and the storage destructor now arm the pool's teardown flag before the lock or
join they would otherwise wait behind. The open request plane carries that flag as its
fence, so a round in flight is refused at its next request, its next retry sleep, or its
next streamed refill — the check lives at the request because a phase is long from making
thousands of requests, not from making one long one, and `CasOperation` already re-checks
admission before every attempt and sleep. Every join and every object's ownership stay
unchanged: the join became short, not optional. A round cut this way is recorded `Stopped`
rather than `Aborted`.

Decommission is deliberately not armed: an already-latched self-remount completes one more
step whose pool-identity probe runs on the open plane, and no arm point early enough to
bound the GC join leaves that step intact.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
…bulk delete)

A 15-minute real-GCS soak measured a sweep round costing 300-617 s per phase, all from
per-object request loops on keys that are write-once by construction — an object whose
only writer mints it once and whose only other mutations are exact-token deletes, so
nothing about it needs re-reading once known. Three loops dominated: `fold_reduce`'s
`GET` volume (2870-3400 per round) turned out to be the sweep's mount-floor probing, not
manifest bodies — `floorForNamespace` reads the mount key of every `/`-prefix of a
namespace for every listed manifest, though the floor is one value per server root;
`manifest_deletes` cost 617 s for 3250 sequential conditional deletes at ~190 ms each; and
`ref_object_cleanup` cost 199-204 s for 512-516 keys at four requests each.

The fix cuts each loop to what the write-once property actually allows: one mount-floor
read per namespace per sweep page (memoized), manifest bodies read only for nominated
orphans and through the existing read-ahead instead of on every listed key, and a new
write-once bulk-delete verb (`removeManyWriteOnce`, backed by `DeleteObjects` where the
store has it) replacing the sequential per-key deletes for owner-removed manifests and for
ref-object cleanup, which now revalidates its cohorts before batching them. None of this
changes what gets deleted or when a namespace or manifest is judged eligible — only how
many requests that judgment costs.

Measured on real GCS: `fold_reduce` 300-380 s -> 2-5 s; `manifest_deletes` 617 s -> 2 s on
a 1506-key round; `ref_object_cleanup` 204 s -> under 1 s.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
…own writes to a hot control object

Every `CREATE TABLE`/`DROP TABLE` on a content-addressed disk mutates one pool-wide object,
`cas/ref_catalog`, through a conditional write. Measured on ten parallel stateless jobs:
`DROP TABLE` p50 2.4 s, p90 11.9 s, max 34.7 s; 113 `PreconditionFailed` in 80 s from 53
threads; the losing writer alone racking up 35 attempts with gaps growing to the 5 s cap;
one `DROP` losing eight races in a row for 15.4 s. `CasRefCatalog::casUpdateImpl` starts
every write with a `GET` and paces a lost race with `Retry::backoff`, a schedule shared with
transport faults — a writer that has lost several races sleeps for seconds while a fresh one
starts at zero, so the oldest loser is the least likely to win next. Worse, every writer in
one process races every other writer in the *same* process: compare-and-swap is only needed
against other servers, so every intra-process race is pure waste, each costing a `GET`, a
refused `PUT`, a resolve `GET` and a sleep.

`CasHotKeys` sits above the request engine as one FIFO ticket per pool and key: writers to
the same hot object queue instead of racing, their conditional writes are combined into one
physical attempt where safe (as-if-serial semantics, a `Conflict` cascade on a lost race so
combined members see the answer a serial retry would have given them), and a last-known-
object cache lets a lane holder skip the leading `GET` under one rule. Losing a race against
*another server* still paces with a flat jitter, not the transport-fault backoff. The GC
erase over `ref_catalog` (`deleteCompletedRemovingAtSnapshot`) becomes the lane's first
caller, and the pool owns the lane. This is phase A only — combining, spacing, the clamp
and moving the GC erase itself onto the lane in full are follow-on work; the design and its
34 review revisions are recorded separately.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
…ed locals

Several gtest fixtures declared a test-only backend hook or fault clock as a local, captured
other locals in it by reference, and then let the store (declared before the hook) call the
hook again during its own teardown — after the captured locals had already gone out of
scope. Under ASan this is a use-after-scope: the store's destructor writes a "farewell"
record that can invoke a still-armed hook whose captured references are already dead.
Fixed by declaring the test clock/hook before the store that keeps calling it (two
transient-round tests, the straggler-epoch test), and by clearing the checkpoint-advance
recovery test's backend hook before the locals it captures die. A separate scripted S3
client fix allocates its response body with `Aws::New`, matching how the SDK actually frees
it, instead of a mismatched allocator. Also corrects two suites that had started asserting a
schedule the engine never promised.

Also: the stateless CAS lanes now run the GC scheduler every 20 s instead of every 5 s,
matching the interval those tests actually need.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
`S3ObjectStorage::getSingleAttemptClient` (`SingleAttemptRetryStrategy`, `max_retries=0`)
and the write path carrying `WriteSettings::object_storage_retry_profile == SingleAttempt`
belong to the CAS control plane's conditional writes: a failed attempt there is not the
final answer — `CasOperation::writeLoop` resolves the outcome by a read and reissues — yet
two upstream sites logged it at Error as if it were terminal: `Client`'s network-error
handler and the non-412 `S3Exception` site in `WriteBufferFromS3`. Both now log at Debug
when the client carries `SingleAttemptRetryStrategy` or the write carries the
`SingleAttempt` profile; an ordinary client configured with zero retries by a user setting
(no outer loop resolving it) keeps logging at Error, since for that caller the failure really
is final. The neighbouring 412 (`isPreconditionFailedError`) branch drops from Info to
Debug for the same reason: a conditional write losing its precondition is the caller's
expected answer, not an operator-facing event.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
… gtests

`SnapshotPublisherLatchedAcrossChunks` had two independent, reachable races. With
`snapshot_log_count_threshold` at 0 (only reachable in this test), `precommitAdd`'s
post-commit trigger dispatches a background publisher whose capture can still be in
flight when `promote`, moments later, becomes lane leader and moves the lane to
`Writing` — a lost race then backs the publisher off, and this pool's frozen
`boot_ms_fn` never advances past that backoff deadline, poisoning every later dispatch
on the namespace for the rest of the test. Fixed by driving `precommitAdd`/`promote`
directly instead of through the shared `publishEmptyPart` helper, draining and
explicitly publishing between the two commits. Separately, the carve hook gated the
leader on the publisher reaching its blocked `PUT`, but under contention the dispatch's
own scheduling delay could outlast that wait's bound, letting a leader released by
timeout (not by the capture it meant to prove) start chunk 2 before the publisher
captured — fixed by gating on the publisher's capture instead, which is causally prior
to the `PUT`. Verified with 20 isolated `gtest_repeat` iterations and two full `CAS*`
gates (2437/2437 each), reproduced only under CPU contention after isolated repeats
alone did not reproduce it.

Separately: a fatal `ASSERT_*` between launching an `AppendCaller`/`Caller` thread and
its explicit `join()` left `TestBody` with the thread still joinable, and
`std::thread::~thread()` on a joinable thread calls `std::terminate`, aborting the whole
`unit_tests_dbms` binary and discarding every test scheduled after it. Both structs now
join in their destructor if still joinable, so a failed assertion costs one test instead
of the whole gate.

Also: `CASDetachedWork` now stops and drains its detached publisher before the locals
its hooks read go out of scope (ASan stack-use-after-return on `fake_boot` via
`boot_ms_fn`), the same class of bug as the earlier test-hook lifetime fixes.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0115C2huxSQJkqDDV24h4JEi
Signed-off-by: Mikhail Filimonov <mfilimonov@altinity.com>
@filimonov filimonov changed the title cas improvements CAS improvements Sep 4, 2026
@filimonov
filimonov marked this pull request as ready for review September 4, 2026 23:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

2 participants