fix(memory): add atomic owner-scoped idempotent write receipts

[loveRhythm1990]

Problem and behavior

Astra issue matrixorigin/Astra#996 needs partial-stage retries that survive process death and lost HTTP acknowledgments. Ordinary store uses a fresh UUID plus approximate/vector deduplication; concurrent requests can create duplicates and similar content can reuse another operation's session metadata. That is insufficient evidence for a durable archive-before-purge decision.

This adds an optional, atomic single-memory idempotency protocol:

• POST /v1/memories/idempotent requires an exact operation key and retains existing authentication, scoped memory-write permissions, branch/main guards, sensitivity checks and per-user SQL routing.
• The authenticated owner, branch and key derive a deterministic receipt. The SQL primary key arbitrates concurrent inserts; matching replays return the actual persisted record, while conflicting payloads and inactive receipts fail. Only the insert winner reports a new memory and schedules entity extraction.
• Ordinary single-store requests remain compatible and may opt in with a key. Batch store rejects keys before mutation. The receipt lasts while its record/tombstone is retained; physical erasure ends that lifetime.

The dedicated route makes old backends fail before a write, so Astra must not fall back to ordinary store after a missing or malformed acknowledgment. Deploy this Memoria change before the matched Astra durable-finalization client
in matrixorigin/Astra#1005. Both PRs are drafts for review.

Review follow-up

Addresses the review comment:

• Ordinary vector dedup excludes receipt-bearing rows, protecting them from metadata refresh and superseding. The public HTTP regression uses deterministic embeddings and verifies that ordinary-row dedup still works and the original keyed receipt remains replayable.
• Fingerprinting explicitly sorts nested JSON objects. The current dependency graph does not enable serde_json/preserve_order, so conversion through json! already sorts HashMap keys; reordered multi-key/nested metadata is now covered by regression tests and remains stable if that feature is enabled later.
• API memory responses strip the reserved fingerprint while preserving caller metadata; single/batch writes and structured filters reject the reserved field. Tests cover response omission and these rejection paths.

Validation

Passed against newly created, isolated local MatrixOne databases and synthetic identities:

• cargo test -p memoria-service --lib idempotency_tests (1 test)
• cargo test -p memoria-service --test idempotent_store_db_it -- --test-threads=1 (1 test; 8 concurrent replays, conflicting content/session, owner separation, soft-delete retry)
• cargo test -p memoria-api --test idempotent_store -- --test-threads=1 (1 public HTTP test; repeated receipts, missing/conflicting keys, authentication, owner separation and batch rejection)
• cargo clippy -p memoria-api -p memoria-service -p memoria-storage -p memoria-core --lib --test idempotent_store --test idempotent_store_db_it -- -D warnings
• git diff --check

Broad all-target Clippy also exposes pre-existing warnings in subject_id_filter.rs, governance_pool_budget_e2e.rs and scheduler lib tests; these unrelated files are unchanged. No QA/production writes or live provider calls were used. No schema migration is needed.

Host review used the open-code-review delegate skill: all 9 selected Rust files reviewed; 2 documentation files read separately; 0 skipped, 100% coverage of this change. Documentation describes the API, deployment dependency and deletion limits.

Review follow-up: resolved branch receipts

Fixed the blocking branch-scope finding from the exact-head review. Keyed writes snapshot the effective checkout (or normalized explicit branch), resolve its table once, and retain the routed SQL store/table for receipt lookup, insertion and conflict replay across embedding. An omitted branch and the same explicit branch now share one receipt; invalid branches fail closed. Clients requiring a fixed retry scope must send it explicitly; Astra #1005 pins its durable reads, writes and cleanup to main.

Validation: the public authenticated HTTP tests pass (2 tests), including checked-out/explicit aliases on both keyed endpoints, branch-local row counts, checkout during a semaphore-paused deterministic embedding, cross-branch key separation and missing-branch rejection. The existing eight-way concurrent replay/conflict/deletion DB regression and receipt identity unit test pass. Clippy passes for API/service/storage/core libraries and both idempotency integration targets with warnings denied. All database tests used an isolated local MatrixOne database and mock embeddings.

[loveRhythm1990]

Review

Is the issue real?

Yes. Looking at the pre-existing single-store path (memoria/crates/memoria-service/src/service.rs, the vector-dedup branch around line 1486+): without an idempotency key, every store call generates a fresh Uuid::now_v7() and relies on find_near_duplicate (an approximate, L2-threshold cosine match scoped only by user_id + memory_type + subject_id, not session_id). Two concrete gaps follow directly from that:

• If a retried write's embedding doesn't land inside the L2 threshold (e.g. slightly different content normalization, or just embedding non-determinism), the retry inserts a genuine duplicate row.
• If it does land inside the threshold, the code returns/updates the other call's record regardless of whether session_id matches — so a retry from session A can be folded into a record that actually belongs to session B's write. The existing Chinese comments around that branch (store.rs dedup block) already acknowledge this is a non-atomic read-then-write with a documented race window.

So the stated problem (no exact-key idempotency, cross-session contamination risk, duplicate-on-retry) is real and verifiable in the current code, not just asserted in the PR description.

Is the fix correct?

The core mechanism is sound: memory_id is VARCHAR(64) PRIMARY KEY (memoria-storage/src/store.rs:1350, :1453), insert_into is a plain autocommitted INSERT (no explicit transaction wrapping), and the idempotent path does early-check → embed/sensitivity → insert → on Err re-check by id → validate_idempotent_receipt. Under MySQL/MatrixOne PK locking semantics, a concurrent loser's insert blocks until the winner commits, then fails deterministically, so the fallback get_from after an insert error is correct and race-free for the "which request wins" question. batch_store rejects keys before any mutation (first statement in the function). Auth/branch guards are correctly extended to the new route (auth.rs:356, lib.rs:129).

However, I found two real correctness bugs in the payload-fingerprinting that backs "conflicting payload" rejection:

1. Non-deterministic fingerprint for multi-key metadata (service.rs:1381-ish, the payload = serde_json::json!([...]) block that feeds the SHA-256 in idempotency/hash). extra_metadata is typed std::collections::HashMap<String, serde_json::Value> (memoria-api/src/models.rs), not serde_json::Map/BTreeMap. Serde's blanket impl serializes a HashMap in its internal iteration order, which is per-thread-seed-dependent and not guaranteed stable across requests landing on different tokio worker threads. Any idempotent store call whose extra_metadata has 2+ keys can therefore get a different SHA-256 on a legitimate retry, and validate_idempotent_receipt (service.rs:~3654) will wrongly reject it with 422 "already used for another payload." This directly undermines the retry-safety guarantee the PR exists to provide — Astra's durable finalization payloads (scene/agent metadata) are exactly the multi-key case. None of the new tests (idempotent_store.rs, idempotent_store_db_it.rs) exercise extra_metadata with more than a trivial shape, so this doesn't surface in CI. Suggest canonicalizing (e.g. collect into a BTreeMap<String, Value> or sort keys) before hashing.

2. Receipt hash field isn't protected from the ordinary dedup path. find_near_duplicate (memoria-storage/src/store.rs:5136) has no concept of idempotently-created rows, and the "same content" branch of the ordinary (non-idempotent) store flow calls update_extra_metadata (store.rs:5335), which does a full SET extra_metadata = ? replace, not a merge. If a later plain /v1/memories call (no key) happens to vector-dedup onto an idempotently-created memory — plausible in the same user/type/subject space this feature targets — and supplies any extra_metadata, it silently wipes the _memoria_store_request_sha256 field (service.rs:1374/:3622). A subsequent legitimate replay of the original idempotency key then permanently fails validate_idempotent_receipt (hash now absent), even though the original operation never changed.

3. Minor hygiene issue: _memoria_store_request_sha256 lives in the same extra_metadata map that's documented as transparent passthrough returned verbatim to callers and filterable via extra_metadata_filter (models.rs:26-29). It leaks into API responses and into client-facing structured queries. Consider a dedicated column, or stripping it from MemoryResponse before serialization.

None of these three issues invalidate the overall design — PK-arbitrated insert as the concurrency primitive is the right approach — but (1) and (2) are concrete ways the idempotency guarantee can silently break in exactly the scenarios this PR is meant to harden, and should be fixed before Astra depends on this contract.

P1 — bug (1) alone means the idempotency guarantee this PR exists to provide can silently fail for any retry with multi-key extra_metadata, which is the primary Astra use case described in the PR body. Should be fixed before this is relied on for durable-finalization retries.

[aptend]

Deep review of exact head f37dd5409a6dd9301766b6f2a5d4fe152e65d13b, base/merge-base 4c99ae4e4ffabb5f2f0f24c5121e0f99c458674e. Read the full eleven-file diff, complete review/comment/thread history and all three commits in the series, including the follow-up against the prior feedback; traced auth/owner routing, branch resolution, primary-key insertion/replay, tombstone behavior, approximate dedup and response/filter consumers.

The earlier concerns are closed: recursively sorted fingerprints, protected reserved metadata, response omission and exclusion of receipt rows from ordinary dedup are present and the public regressions pass. The original HashMap concern is not repeated as an unfixed defect; the dependency graph's default JSON map was already ordered, and explicit recursive canonicalization now also protects preserve_order consumers.

One independently reproduced blocking idempotency defect remains: omitted branch hashes as main, while storage follows the checked-out branch. The same owner/key/payload in the same actual branch can thus acquire two different persisted receipts depending only on whether the branch is explicit. See inline reproduction.

Validation:

• Complete repository Unit Tests lane (memoria-api, memoria-core, memoria-service, memoria-mcp libraries): 241 passes, zero failures/ignored, Rust 1.94.0, locked dependencies. The new identity unit also passed separately.
• Unmodified committed service DB test and public HTTP test both passed twice against a fresh, test-owned MatrixOne container with synthetic identities/databases; no QA/production service or live embedding/LLM provider was used. Server: cached official 4.2.1-hotfix-20260916, image digest sha256:2f4c6ecfe084912474a28ebf2adb932876abbbb6b123bc17677ed65668e1ea4e. These cover eight concurrent replays, conflicts, owner separation, deletion, ordinary dedup protection, reordered nested metadata and reserved-field rejection.
• Additional public authenticated HTTP branch counterexample used the existing multi-db fixture and deterministic mock embedder. Two isolated owners/rounds each produced two different IDs and two active rows in reviewb, zero in main for the same key/payload; explicit-branch replay returned the same explicit ID as a positive control.
• Exact-head Unit/DB/Clippy CI checks are green; the DB job log independently confirms both new integration tests actually ran and passed. Those tests do not exercise omitted-versus-explicit checked-out branch identity.
• Initial current-source MO harness attempts failed on temporary build capacity or unsupported SQLx default handshake character set 224 in an admin fixture connection. They are environment/fixture diagnostics, not Memoria product findings or green HTTP evidence; the final official-container runs above used the original assertions without production/fixture patches.

Please derive receipt identity from the same frozen effective branch/table used for lookup and insert, or require an explicit branch for keyed writes and fail closed on omission. Add checked-out/explicit alias and checkout-during-await coverage. Test-only probes and the disposable server will be removed after submission.

[aptend]

APPROVE — deep re-review of exact head ed65b7a7ce0e259e1d27b096319e4290f363ff45, base/merge-base e3665e8def58ed5f58b109fdd4594feed9df0398, compared with my previous reviewed f37dd5409a6dd9301766b6f2a5d4fe152e65d13b.

The branch-scope P1 is CLOSED in the implementation. Keyed writes resolve the effective checkout/normalized explicit branch once, derive the receipt from that same branch and retain the routed SQL store/table through lookup, embedding, insert and conflict replay. Omitted and explicit aliases now use one identity and one destination; a checkout during await cannot redirect the operation, and missing/deleted branches fail closed. No remaining blocking finding established.

Read all eleven current changed files, the incremental repair and all review/inline/author conversation/thread history (one resolved thread). Reconciled upstream-only metrics/MCP changes separately. Traced authentication/scoped owner/group-main guard, key validation and canonical payload hashing, PK-arbitrated winner/loser paths, sensitivity/embedding error/cancel, soft-deleted receipt behavior, ordinary dedup exclusion, reserved metadata/filter protection, response omission and batch rejection. The frozen Arc/store/table is request-owned, not a connection held across embedding; only successful insertion schedules/report new-memory work. Retention/deletion limits and explicit-branch advice for retries across separate checkout changes are documented.

Verified/reused primary CI evidence from run 38617576754, Rust 1.85.0:

• Unit Tests: api 106 + core 22 + mcp 44 + service 69 = 241 passing library tests, zero failures/ignored.
• DB job 116037610774, original logs: owner_scoped_store_replays_real_receipts_and_rejects_missing_keys and receipts_follow_resolved_branch_and_checkout_cannot_retarget_inflight_write both PASS (2/2, zero ignored). Read their actual assertions: both keyed endpoints, padded explicit branch, one branch-local row, deterministic semaphore-paused embedding/checkout, frozen destination, cross-branch separation, missing-branch rejection, metadata protection and auth/conflict controls.
• Committed concurrent_replays_have_one_receipt_conflicts_do_not_mutate_and_deletion_is_final PASS (1/1, zero ignored): eight concurrent callers, owner/session/payload conflicts and inactive receipt rejection.
• Check & Clippy succeeds; source whitespace check and isolated-worktree source state are clean.

CI checks out synthetic merge 9ad247dbdc9261862362a098eb194edf492d449d, not the literal head commit. I verified its parents and identical full source tree da0812c5793882194525f2727c9cb8a91609eecb against the exact review head before reusing the results. No fresh local Rust/DB rerun or live-provider/QA/production write is claimed. The earlier independent failed branch counterexample and original tests remain historical, not relabeled current runs. Head/state rechecked immediately before submission.