Skip to content

Memory: budget CAS retry lacks spin_loop backoff — contended allocations burn pipeline slots #399

Description

@EnRaiha

Version / build tested against

origin/main @ e235fe5 (2026-09-29)

Deployment mode

Origin — single node (local)

Engine(s) involved

Not engine-specific / unsure

Summary

The nodedb-mem global budget counter is updated with a tight compare_exchange_weak retry loop whose retry arm has no std::hint::spin_loop() (or other backoff). Under bursty multicore allocation this burns pipeline slots on cache-line contention.

Steps to reproduce

Static: inspect the retry arms in nodedb-mem/src/scoped_budget.rs / budget.rs — Err(_) => continue / Err(actual) => current = actual with no spin_loop(). (runtime characterization pending.)

Expected behavior

Retry arm yields with std::hint::spin_loop() between CAS attempts.

Actual behavior

Tight retry loop without a backoff hint.

What actually happened? (severity facts)

  • Acknowledged/committed data was lost, corrupted, or silently wrong
  • The server crashed, hung, or failed to start
  • A security or isolation boundary was crossed
  • Core functionality is broken with no acceptable workaround
  • A workaround exists

Proposed severity

SEV-4 — Low: contention/perf hardening only.

Reproducibility

Always — every attempt (code path constant)

Last known-good version / commit (if a regression)

(unknown / not a regression)

Environment & logs

Linux x86_64; Verified by static code reading at the pin above; runtime reproduction pending.
Code references:

  • (see prior-art line below)

Before submitting

  • I searched existing issues and this is not a duplicate. (searched spin_loop/governor: none)
  • Reproduced on a released tag or current main build (pending) — code path verified at e235fe55c.
  • This is not a security vulnerability.

Additional evidence (origin/main @ e235fe55c)

  • What: The global-ceiling credit in ReserveScope::try_credit_global retries on CAS failure with a bare continue, with no processor backoff hint. The same shape appears in the engine and scoped-budget counters. There is no std::hint::spin_loop() anywhere in nodedb-mem.
  • Where: nodedb-mem/src/reserve_scope.rs:71-96 (global.allocated CAS; failure arm Err(_) => continue); nodedb-mem/src/budget.rs:101 (engine/per-db Budget::try_reserve); nodedb-mem/src/scoped_budget.rs:48 (ScopedBudget::try_reserve).
  • Evidence: rg -n 'spin_loop|std::hint' nodedb-mem/src/ → no matches. rg -n compare_exchange_weak nodedb-mem/src/ → reserve_scope.rs:81, budget.rs:101, scoped_budget.rs:48, budget.rs:28, budget.rs:74.
  • Impact: Low. The loop is bounded (returns GlobalCeilingExceeded rather than spinning forever), so this is a micro-optimization, not a correctness defect. The allocated atomic is process-global and shared by every engine/database/tenant, so it is the governor's hottest contended cache line; on a retry the core issues another RMW immediately, which lengthens the contention window under bursty multi-core allocation.
  • Fix: Add std::hint::spin_loop() in the CAS failure arm before retrying (and in the peer loops in budget.rs / scoped_budget.rs). Measure before/after under a multi-thread allocation benchmark; if no measurable delta, close as won't-fix.
  • Prior-art: no open issue matches (searched: spin_loop, governor, CAS).
    Why: a tight compare_exchange_weak retry loop on the global counter burns pipeline slots under multicore contention; the retry arm has no std::hint::spin_loop().
    Steps to verify: rg -n "compare_exchange_weak" nodedb-mem/src -> the retry arms in scoped_budget.rs/budget.rs lack spin_loop(); add and re-run the nodedb-mem tests (cargo test -p nodedb-mem).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions