Repository navigation
feat: add Redis-backed editor data storage for high availability - #46
nvanlaerebeke wants to merge 62 commits into
Conversation
|
Hi, this is an interesting PR and converges with #44. I feel 44 has more depth, but this is wider in feature set. They also use differing libraries, node-redis and ioredis. Being as your submission is AI, I would take the opportunity to run AI to compare both PRs. My questions are around depth, features and libs. I will update here once I have done that. |
|
FYI, the reason I didn't use ioredis was that the node-redis is I think the "more" official one. Updating that package to 5.x also gives support for redis sentinel etc and I thought it wasn't productive to have multiple packages that basically do the same. I'll see if I can bring this PR closer to #44 |
|
Hi @nvanlaerebeke I would agree having multiple packages doing the same is not productive. I think I would also agree with you about a shift to node-redis and would be more of the mind to change 44 to use that than vice versa. |
|
@nvanlaerebeke Would it be useful for you if I run a comparison and share it? (44 and 46 - they point at different things, but might be useful and I can burn some tokens so you don't have to) |
|
@nvanlaerebeke @MonaAghili — comparison of #46 against #44, read at ConvergentFour non-obvious
A clean-room derivation from DivergentSurface. #46 implements the full surface including #44's partial state is not merely incomplete, it is harmful: Test infrastructure. #44's redis suite does run in CI inside the existing unit job against an ephemeral server, and #44 has additionally been exercised against a real multi-node cluster under sustained multi-replica load outside CI. A cluster CI job establishes that the code runs correctly on a cluster; a field run establishes that it survives one. Both are needed. Failure policy. #46 throws from every method with no call site changed (16 files, none of Findings1. Both cross-document indexes are on one slot. 2. The command timeout outlives the transport. 3. Sentinel deployments run standalone silently. The orchestrated entrypoint emits sentinel configuration only under The reverse applies to #44: the standalone entrypoint writes credentials only under 4. Destroying the client on timeout is correct and #44 has no equivalent. It is the only mechanism that turns a black-holed connection into a reconnect: when a peer dies without TCP teardown — SIGKILL, an idle drop in a NAT or load balancer, a partition — there is no ioredis's 5. Client librarynode-redis, per Redis's own recommendation for new projects. Version: 6.x, with one decision to make explicitly. Current is 6.2.1; the 5 line is at 5.12.1; 4.7.1 is tagged
Cluster options belong in Command resend. node-redis does not replay a command already written — on socket close, sent commands reject rather than re-running — so the "was the lock actually taken?" hazard is closed on standalone and cluster without an option, where ioredis needed one. The exception is sentinel. This lands harder on #46's surface than #44's. A re-run is harmless where the operation is idempotent, and most of #44's ten are — a re-granted re-entrant lock, a repeated presence write. On the full surface it is not: Evidence. Existing field measurements were taken against ioredis on a real cluster and do not describe a node-redis build. Equivalents are now identified for the offline queue, the resend behaviour and the timeout, which scopes re-measurement to Cluster readiness has no client-side signal: Config surface. #46 removes the Open, not delivered
Neither implementation has been read line by line beyond the areas above, and #46's cluster job has not been run here, so the findings are for verification rather than acceptance. |
|
Independent deep-dive on #46 against Where this overlaps with the comment above, I've marked it; the rest is new. SummaryThe hard behavioral work is largely right: lock re-entrancy, the numeric unlock enum, But five separate things block merge, three of which aren't in the discussion yet: Blockers1. Issue #267 itself is not demonstrably fixed. The new module is split into six files under 2. // editorStat.js:228-230
EditorStat.prototype.deleteKey = async function (key) {
await this._command(['DEL', key]);
};
3. Force-save Lua scripts This round-trips the entire 4. Sentinel support is silently inert. The orchestrated DocumentServer entrypoint writes sentinel settings under 5.
Also worth flagging (not blocking, but real)
Test gaps
Bottom lineCompetent and the most complete of the candidates — full |
|
@nvanlaerebeke @MonaAghili I will run a check against Mona's deep dive - the referenced issue is not actually in this repo. It is here: Euro-Office/DocumentServer#267 @nvanlaerebeke thanks for reacting so quickly to replies here, really appreciate it |
|
@nvanlaerebeke @MonaAghili — both reviews verified against If you are working on this tonightYou said you had pulled #44 and were applying this on top. Four things that are worth having before you start, rather than after. Sign off while you rebase. DCO is failing on all four commits and it is the only completed check on this PR. Client library is settled: node-redis. #44 moves to it. You should not end up carrying both clients through the rebase. Do not spend time on these — each is something one of the two reviews above could reasonably send you doing, and none of them is needed:
CI is now released. Checks on this PR had never run — they were held behind the fork-approval gate. Both are approved and executing against If you only get to a few things: TL;DR
Corrections — my earlier comment
Unchanged: slot 1073 (re-confirmed by Corrections — the review aboveB1, #267's stack fails in Neither PR is scoped to #267. A complete, tested Redis backend answers it; the packaging gap is fixed on its own terms. What belongs in this PR is the demonstration: boot a packaged DocService with (A bare W4, dependency growth — inverted. B3, cjson — three mechanisms reproduce, none has a live trigger. Empty-array-to-object and 16-digit truncation reproduced directly, with identical output on Redis 8.10, B2, B5a, W6, installed 4.7.0 — local to that checkout; the shrinkwrap pins 6.2.1 and the workflow runs Confirmed as stated: B5b Contradictions between the two reviews
Remaining workDefects:
Not demonstrated rather than broken: #267 end-to-end (boot a packaged DocService with Client librarynode-redis. Redis recommends it for new projects, two packages doing the same job is not worth carrying, and this PR is already there. #44 moves. The rebase should not end up carrying both clients. Open under it: node-redis 6 defaults to RESP3 ( Open: fail-open versus fail-closed
A — throw from every method (this PR today). Uniform, no call-site changes, and safe where a plausible-looking value is dangerous: B — return decided values (#44, for its ten). Matches existing call-site expectations; denial is safe for the lock methods and a delayed save costs little. Against: for readers there is often no safe value — C — per-method. Most work, and the answer space is three-valued: deny (lock methods); throw or return a distinct sentinel ( #44's existing policy covers the ten methods it implements, so that pattern is safe for those during the rebase. The methods beyond them are where this bites. Views welcome, including on whether per-method is worth it for a first enablement or whether uniform throw is an acceptable start with per-method to follow. |
|
@j-base64 just to add you here |
|
CI results on
The cluster failure is the harness, not the code. The nodes start correctly via Add This job had never executed before today, so it is not a regression; the fork gate meant nothing had surfaced the missing tool. On the pkg question: the Amendment to my first comment: I described this cluster job as better than anything on our side. The design still is, and the finding does not change that — it had just never run. DCO remains the only other red, and |
|
Merged Re-approved the checks against the new head. |
913378a to
ee22902
Compare
|
I suggest using the following configuration structure to support:
The structure follows the current {
"services": {
"CoAuthoring": {
"redis": {
"name": "redis",
"prefix": "ds:",
"host": "redis.example.com",
"port": 6379,
"options": {
"username": "redis-user",
"password": "redis-password",
"database": 0,
"socket": {
"tls": true,
"rejectUnauthorized": true
},
"commandOptions": {
"timeout": 30000
}
},
"optionsCluster": {
"rootNodes": [
{
"url": "redis://redis-cluster-0.example.com:6379"
},
{
"url": "redis://redis-cluster-1.example.com:6379"
},
{
"url": "redis://redis-cluster-2.example.com:6379"
}
],
"defaults": {
"username": "redis-user",
"password": "redis-password",
"socket": {
"tls": true,
"rejectUnauthorized": true
}
},
"commandOptions": {
"timeout": 30000
}
},
"optionsSentinel": {
"name": "mymaster",
"sentinelRootNodes": [
{
"host": "sentinel-a.example.com",
"port": 26379
},
{
"host": "sentinel-b.example.com",
"port": 26379
}
],
"database": 0,
"nodeClientOptions": {
"username": "redis-user",
"password": "redis-password",
"socket": {
"tls": true,
"rejectUnauthorized": true
}
},
"sentinelClientOptions": {
"username": "sentinel-user",
"password": "sentinel-password",
"socket": {
"tls": true,
"rejectUnauthorized": true
}
},
"commandOptions": {
"timeout": 30000
}
}
}
}
}
}The example shows all supported topology blocks for reference. In an actual deployment, only one topology should be configured; the other blocks should remain empty.
Topology selectionThe current implementation selects the topology as follows:
Configuration details
Legacy configuration
The older/intermediate fields The default configuration is defined in The normalization logic is implemented in |
|
Correction to my comment above, on the mechanism rather than the conclusion. I wrote that with force-save state per replica, "every replica compares against a record another replica armed". That is backwards. Each replica compares the client's document-wide change index against its own force-save record, and only saves handled by that replica ever advance it — so a participant's replica arms a record early, every later save goes through a different replica, and the local record never moves. The conclusion is unchanged: sharing the locks while leaving force-save per-replica is worse than sharing neither. The deployment that reported this had the mechanism right before we did. Nothing in this PR is affected — the surface is already complete, which is the right answer under either reading. |
|
Really nice work @nvanlaerebeke 🙌, and thanks for pushing this forward. Having the full surface plus standalone/cluster/sentinel in one place looks like an interesting base to build on. With that in mind, a few things that might be worth a look:
Keen to see where this goes 😊 |
|
Thank you everyone for the positive feedback and for taking the time to review the implementation in such detail. I’m still learning this part of the codebase, so I’ll try to take this as far as I can with my current knowledge. I also appreciate any further guidance to get this merged. I'm pushing euro office at my work place as a replacement for office for the web. Regarding the Redis configuration I previously suggested, the DocumentServer entrypoints expect the What would be the preferred way to handle this?
I'm not a fan of "3", redis is currently not part of the codebase at all so it's better to start without any "legacy" as there should be no deployments with that configuration as those won't start due to the missing storage provider. |
|
I wanted to look into the failure behavior, for completion I've added a list of all the methods and the suggested solution.
|
|
Because there is a lot of information in the PR I generated a "current status" about what's done and still pending to get a better overview of the current state. Current statusThe Redis editor-data implementation has progressed significantly since the initial review. The recent work has focused primarily on standalone Redis and Redis Cluster support. Redis Sentinel remains an open workstream and has deliberately not been treated as complete yet. Completed
Sentinel statusSentinel is not considered complete yet. The current branch contains Sentinel option normalization and bounded reconnect behavior, but the remaining integration work includes:
This is the main topology-related item still open. The corresponding DocumentServer entrypoint work is being handled separately. Remaining Redis workThe main remaining work inside this PR is:
Intentionally not required for this PR
The Redis and Redis Cluster implementation is substantially further along. Sentinel integration and the remaining failure-policy decisions are the main items still requiring focused work. |
|
Hi @nvanlaerebeke thanks for your contribution, and great work! I've done another review Reviewed at HEAD: 1. Executive summaryThe implementation is in materially better shape than the previous review round found. Every The mechanism that fixes #267 is sound. Two things block merge. Neither is a code-quality problem:
Beyond those, the most substantive remaining concerns are a non-retry-safe 2. What the requirement actually isIssue #267 is a packaging defect, not a feature request:
The acceptance bar is therefore:
Everything else in this review — failure semantics, distributed correctness, GC, Sentinel — is Why the module was missing, and why it is now found
"pkg": {
"scripts": [
"./sources/editorDataMemory.js",
"./sources/editorDataRedis.js", // listed before this PR — the file just did not exist
"./sources/pubsubRabbitMQ.js",
...
]
}The module is loaded dynamically from a config string, so pkg's static analysis cannot reach it from
// DocService/sources/editorDataRedis.js
'use strict';
// Compatibility entry point for the pkg configuration and existing consumers.
module.exports = require('./editorDataRedis/index.js');and the chain from there is all static literals: Both build paths use the same manifest-driven invocation, so the fix covers the
Conclusion: the packaging fix is correct by construction. It has not been demonstrated by an 3. Method
The specification for this review is issue #267. 4. BlockersBlocker 1 — The packaged smoke test has never executedLocation: Requirement violated: #267 is a packaging defect; only a packaged binary starting with Evidence (from
Job breakdown for the failing run ( The Cluster job failed on infrastructure, not on product code: 13 commits have landed since that run, none of them tested:
Impact: The PR's central claim is unevidenced. The known-failing Cluster job is plausibly fixed Why current tests don't catch it: They have not run. Suggested fix: Approve and run both workflows on Confidence: High — directly verified against the GitHub Actions API. Blocker 2 — Sentinel config key mismatch silently disables SentinelLocation: Invariant: A deployment that configures Sentinel must get Sentinel, or must fail loudly. What the PR changed in "options": {},
"optionsCluster": {},
- "iooptions": {
- "lazyConnect": true
- },
- "iooptionsClusterNodes": [],
- "iooptionsClusterOptions": {
- "lazyConnect": true
- }
+ "optionsSentinel": {}What the module reads: // base.js:83
const cfgRedisOptionsSentinel = cfgRedis.get('optionsSentinel');
// base.js:216-219
function hasNodeSentinel() {
const options = cloneConfig(cfgRedisOptionsSentinel) || {};
return Object.keys(options).length > 0;
}What the orchestrated DocumentServer entrypoint actually writes: "redis": {
"name": "'${REDIS_CONNECTOR_NAME:-redis}'",
"host": "'${REDIS_SERVER_HOST:-localhost}'",
"port": '${REDIS_SERVER_PORT:-6379}',
"options": { "user": "...", "password": "...", "db": "..." },
"optionsCluster": { '${REDIS_CLUSTER}' },
"iooptions": { // <-- Sentinel lives HERE
"sentinels": '${REDIS_SENTINEL}'
"name": "'${REDIS_SENTINEL_GROUP_NAME:-mymaster}'",
"sentinelPassword": "'${REDIS_SENTINEL_PWD}'",
"username": "...", "password": "...", "db": "..."
}
}Failure scenario:
Impact: A Sentinel-configured deployment gets a single-node connection with no master discovery
Why current tests don't catch it: '...,"redis":{"host":"127.0.0.1","port":%s,"prefix":"pkg-smoke:%s:","optionsCluster":{},"optionsSentinel":{}}...'so the smoke test never sees entrypoint-generated config. The Sentinel unit tests in Answering the self-consistency question directly: the companion DocumentServer change is Suggested fix (pick one):
Confidence: High — read directly from the entrypoint source and the module's config reads. 5. Major issuesMajor 1 —
|
| Caller | Context |
|---|---|
DocsCoServer.js:1529 |
cleanDocumentOnExit after a successful save |
canvasservice.js:1317 |
the storeForgotten path, whose inline comment still reads "clean redis (redisKeyPresenceSet and redisKeyPresenceHash removed with last element)" |
Assessment: the conditional is genuinely protective — it is what makes the GC claim race safe.
POP_EXPIRED removes a member from the source index at claim time, so a document that gets
re-activated between claim and processing would otherwise be wiped while users are editing it. The
guard prevents that, and presence.tests.js covers it
('document cleanup does not delete presence while a user remains').
The problem is that it is an unannounced contract change: nothing in Readme.md or the code
comments records it, the stale comment at canvasservice.js:1316 now describes behaviour that no
longer happens, and no differential test pins the intended difference against editorDataMemory.
Suggested fix: Document the deviation, update the stale comment, and add a differential case
asserting the intended difference rather than leaving it implicit.
Confidence: High.
Major 3 — Sentinel has no executable coverage of any kind
redisEditorDataTests.yml defines exactly two jobs:
redis-editor-data-tests—redis:7-alpineservice container (standalone)redis-editor-data-cluster-tests— six-node cluster viadocker run
There is no Sentinel job. The six Sentinel tests in connection.tests.js assert the shape of
normalizeSentinelOptions output and the reconnect-strategy curve:
- normalizes native Sentinel options and applies the selected database to node clients
- uses bounded deterministic backoff for Sentinel node reconnects
- rejects incomplete native Sentinel options
- uses the native Sentinel raw-command signature
- uses the native Sentinel batch-command signature
- rejects an initial Sentinel connection failure and clears the failed client
None starts a Sentinel. createSentinel, master discovery, Sentinel authentication, TLS, failover,
reconnect-during-failover and the sendCommand(false, …) signature against a real server are all
unexercised — and ec7900f5, the newest commit in the PR, changes exactly this surface
(disableOfflineQueue, commandsQueueMaxLength, reconnectStrategy, maxCommandRediscovers).
Combined with Blocker 2, the
Sentinel path is both unreachable from real deployment config and untested.
Confidence: High.
Major 4 — EditorStat.deleteKey now really deletes a caller-supplied, unprefixed key
Location: editorStat.js:232-234
EditorStat.prototype.deleteKey = async function (key) {
await this._command(['DEL', key]);
};versus the memory backend's deliberate no-op (editorDataMemory.js:499-501):
EditorStat.prototype.deleteKey = async function (_key) {
//no need
};Reachability: only when process.env.REDIS_SERVER_DB_KEYS_NUM is set, which constructs the proxy
on a separate logical database (DocsCoServer.js:156-158):
let editorStatProxy = null;
if (process.env.REDIS_SERVER_DB_KEYS_NUM) {
editorStatProxy = new editorStatStorage.EditorStat(process.env.REDIS_SERVER_DB_KEYS_NUM);
}All five call sites guard on the method's existence, so there is no TypeError:
| Call site | Argument |
|---|---|
DocsCoServer.js:1531 |
docId |
DocsCoServer.js:2191 |
docId |
canvasservice.js:480 |
docId |
canvasservice.js:495 |
mask.key |
canvasservice.js:1319 |
docId |
Concern: the key is used verbatim and unprefixed — DEL <docId> in whatever logical DB the
env var names. That key format is an upstream-owned convention this repository does not define.
Implementing the interface is defensible; guessing the key layout and issuing DEL against a
possibly shared instance is the risky part, and it is the one operation in the module that can
destroy data belonging to another component.
Note also that with Cluster configured, constructing this proxy with a non-zero database throws at
first use (base.js:426-429), which is correct but deferred to command time rather than construction
time.
Suggested fix: Either keep the no-op (matching the memory backend and the fact that the proxy
scaffolding is inert upstream machinery), or document the key contract and namespace it.
Confidence: High on behaviour; reachability gated by an env var nothing in this repo sets.
6. Verified fixes
Every item below was a finding or topic in the earlier review round and was re-tested against
ec7900f5. All are genuinely fixed. Do not re-report these.
6.1 Stable lock-ID removal — FIXED
-- REMOVE_LOCKS_SCRIPT
for i = 1, #ARGV, 2 do
redis.call('HDEL', KEYS[1], ARGV[i])
end
return redis.call('HLEN', KEYS[1])argsFromObject(locks) produces [id1, json1, id2, json2, …]; the stride-2 loop consumes only the
odd indices, i.e. the lock IDs, and ignores the JSON payloads entirely. Deletion is by hash field
name — the stable ID — not by value.
Checks demanded by the review brief:
| Question | Answer |
|---|---|
| Are lock IDs unique enough? | They are the caller's own hash keys, generated upstream by the SDK; the backend does not invent them. |
Does the ID survive _recalcLockArray? |
Yes. DocsCoServer.js:2391-2425 builds res = {} and assigns under the same lockId; it mutates lock.block.rangeOrObjectId.* only. |
| Can removal delete another lock? | No — HDEL by exact field name. |
| Can a stale owner remove a newer lock? | removeUserLocks (DocsCoServer.js:2249-2262) re-reads getLocks immediately before building toRemove, filtered by lock.user === userId. |
| Are Redis and memory semantics equivalent? | Yes — memory does delete data.locks[lockId] over the same key set. |
| Concurrent mutation | The whole loop is one Lua script, hence atomic. |
One genuinely important detail the fix gets right: in the memory backend _recalcLockArray mutates
the shared data.locks objects in place, so no write-back is needed. With Redis, getLocks
returns freshly decoded objects, so the write-back at DocsCoServer.js:3602
(yield editorData.addLocks(ctx, docId, docLockMod)) is load-bearing. It is present.
6.2 Force-save payload corruption through cjson — FIXED
The payload is never decoded by Lua. STORE_FORCE_SAVE_SCRIPT writes pre-serialized JSON into
discrete hash fields, and START/SET only touch scalars:
redis.call('HSET', KEYS[1],
'time', ARGV[1], 'index', ARGV[2],
'baseUrl', ARGV[3], 'baseUrlDefined', ARGV[4],
'changeInfo', ARGV[5], 'changeInfoDefined', ARGV[6],
'convertInfo', ARGV[7], 'convertInfoDefined', ARGV[8],
'started', '0', 'ended', '0')Verified against the brief's checklist:
- Payload opaque — yes, no
cjson.decode/encodeanywhere inscripts.js. - Only intended scalars change —
STARTsetsstarted/ended;SETsets
started/ended/convertInfo/convertInfoDefined. changeInfobyte-identical — never rewritten aftersetForceSave.convertInfopreserved — with an explicitconvertInfoDefinedflag distinguishingnull
from omitted (editorData.js:254-256).- Arrays stay arrays / large numbers unchanged — because JSON round-trips in Node, never Lua.
- State transitions equivalent —
STARTrequiresstarted ~= '1', matching
if (data.forceSave && !data.forceSave.started).
A bonus improvement over the prior design: time/index are compared as jsonEncoded strings, so
123 and '123' no longer conflate — this is now stricter than before and matches the memory
backend's ===.
Concurrency, as the brief asked:
| Sequence | Outcome |
|---|---|
A and B both checkAndStartForceSave |
Exactly one wins; the loser gets nil. Asserted in process.tests.js:444-449 across two real processes. |
A and B both checkAndSetForceSave |
Guarded by the time/index CAS; a stale caller gets nil. Asserted at process.tests.js:450-460. |
A removeForceSave races B setForceSave |
DEL vs HSET — last writer wins. A stale removeForceSave can drop a newer record; this is the same race the memory backend has, and no caller interleaves them. |
6.3 POP_EXPIRED lost-response behaviour and batching — FIXED
The design is a proper claim / lease / acknowledge protocol with per-claim tokens:
-- POP_EXPIRED: claim() moves a member into the lease ZSET and records the claim token
local function claim(member)
redis.call('ZADD', KEYS[2], leaseUntil, member)
redis.call('HSET', KEYS[3], member, claimId)
table.insert(values, member)
end-- ACK_EXPIRED: only the current claimant may release
if redis.call('HGET', KEYS[2], ARGV[2]) ~= ARGV[1] then
return 0
end
redis.call('HDEL', KEYS[2], ARGV[2])
return redis.call('ZREM', KEYS[1], ARGV[2])Key-order check: _ackExpired(leaseKey, claimsKey, item) passes [leaseKey, claimsKey], so
ZREM KEYS[1] targets the lease ZSET, not the source index. A late ACK therefore cannot delete a
freshly re-added source entry. Claim tokens are monotonic per instance
(${randomUUID()}:${++sequence}, editorData.js:136-139).
Against the brief's exact adversarial scenarios:
| Scenario | Behaviour |
|---|---|
| Worker A claims, crashes, lease expires, Worker B claims, A sends delayed ACK | A's ACK carries the old claimId; HGET mismatch → return 0. B's claim survives. |
| Worker A claims 100, processes 40, crashes | 40 acked and released; the other 60 stay in the lease ZSET and are reclaimed after POP_EXPIRED_LEASE_MS (5 min). |
| A newer expiry written while leased | Reclaim checks ZSCORE KEYS[1] member; if present, it drops the stale claim (HDEL) and lets the newer source entry win. |
| Duplicate processing | At-least-once by design; gc.js operations are idempotent. |
| Multiple GC workers | Claim and index-removal happen in the same script, so no two workers can claim the same source entry. |
| Cluster hash slots | All three keys share the {editor:index} tag. |
gc.js consumes it correctly — the ACK helper degrades safely for the memory backend:
function acknowledgeExpired(editorData, method, item) {
const acknowledge = editorData[method];
return typeof acknowledge === 'function' ? acknowledge.call(editorData, item) : undefined;
}One subtlety worth recording as correct: _popExpired stores claim IDs in a module-level
WeakMap keyed on the decoded item array, and gc.js:143 does expiredKeys.sort(...). Sorting
reorders the array but preserves element identity, so the WeakMap lookup still resolves. Both
gc.js call sites use docsCoServer.editorData, the same instance that issued the claim.
Coverage is real — ten tests in expiration.tests.js, including
'does not let a stale document-presence acknowledgement remove a retry',
'recovers only the unacknowledged document-presence item after a partial batch', and
'recovers document-presence expiration after a client timeout'.
6.4 Packaged editorDataRedis smoke test — EXISTS AND IS CORRECTLY DESIGNED (but unrun)
tests/redis/editorDataRedis.pkg-e2e.sh genuinely satisfies the brief's seven-point bar:
| Requirement | Evidence |
|---|---|
| Builds the pkg binary | e2e.yml:92-100 — pkg . -t linux-x64 per service |
| Uses the packaged binary | pkg-e2e.sh:11 locates */server/DocService/docservice inside the overlay image; the source tree is absent |
Enables editorDataRedis |
pkg-e2e.sh:126 sets both editorDataStorage and editorStatStorage |
| Same runtime path as production | Runs the real binary under NODE_ENV=production-linux, NODE_CONFIG_DIR=/etc/euro-office/documentserver |
| Starts DocService | pkg-e2e.sh:130-134, after supervisorctl stop docservice to isolate the port |
| Verifies health | /healthcheck must return true, then three more probes over ~6 s |
| Proves the backend loaded | grep -q '\[editorDataRedis\]' on the log, plus an explicit MODULE_NOT_FOUND guard |
The [editorDataRedis] grep matches REDIS_LOG_PREFIX (base.js:62) and is reachable because the
test forces log.options.categories.default.level = DEBUG, and _createClient logs at debug.
This is a good test. It has never executed — see Blocker 1.
Note also that it pins optionsCluster/optionsSentinel to {}, so by construction it cannot catch
Blocker 2.
6.5 Process-based differential testing — FIXED, and genuinely multi-process
editorDataRedis.process.worker.js is a real child_process.fork target, not an in-process fake:
| Property | Evidence |
|---|---|
| Independent processes | process.send / process.on('message'); distinct process.pid reported |
| Independent connection state | Each worker constructs its own EditorData + EditorStat at module load (worker.js:9-10) |
| Independent shard IDs | EditorStat.sampleId = ${process.pid}:${randomBytes(12)} (editorStat.js:27) |
| Independent timers | Separate event loops |
| Actual process crash | harness.kill('replica-a', 'SIGKILL') (process.tests.js:521) |
| State after one process disappears | runCrashScenario polls from replica-b and asserts the write survived |
| Cross-process visibility | runCrossProcessRedisScenarios writes on A, reads on B throughout |
| Concurrent operations | Promise.all([...checkAndStartForceSave on A and B]), asserting exactly one winner |
The lost-response injection is real — holdAfterCommand lets the command commit, emits
redis-command-committed, then returns a never-settling promise:
if (message.holdAfterCommand) {
return Promise.resolve(result).then(() => {
send({type: 'redis-command-committed', requestId: message.requestId});
return new Promise(() => {});
});
}What the oracle still does not cover (answering the brief's question directly): runTrace covers
save/auth locks, document locks, addLocksNX, removeLocks, messages, setSaved/getdelSaved, the
force-save lifecycle, the force-save timer, unique users, monthly users and lockNotification. It
does not cover cleanDocumentOnExit, removeAllLocks, removeMessages, removeForceSave, the
shard connection counters, setEditorConnections, shutdown tracking, licence storage or deleteKey.
Presence is necessarily Redis-only (the memory backend is degenerate there) and is covered separately
by runCrossProcessRedisScenarios and presence.tests.js. The whole suite is describe.skipped
under Cluster.
6.6 RESP2 — FIXED, verified against node-redis source
base.js sets RESP: 2 at every client level. I verified propagation in @redis/client@6.2.1
rather than assuming it:
- Cluster —
dist/lib/cluster/cluster-slots.js:550:#createClientpasses
RESP: this.#options.RESPto every node client, and line 465 does the same for the discovery
client. SettingRESPat the top level innormalizeClusterOptionsand stripping it from
defaultsis therefore correct, not a bug. - Sentinel —
dist/lib/sentinel/index.js:600-601, with the source comment
"first take the globally set RESP", thenRESP: this.#RESP.
This matters because ADD_LOCKS_NX_SCRIPT returns {conflict, HGETALL} and decodeHash expects
flat RESP2 arrays.
6.7 Sentinel command replay bound — FIXED
REDIS_SENTINEL_MAX_COMMAND_REDISCOVERS = 0. In sentinel/index.js:
this.#maxCommandRediscovers = options.maxCommandRediscovers ?? 16; // line 561
...
if (++count > this.#maxCommandRediscovers) { ... } // line 720With 0, the first failure makes ++count === 1 > 0 true and the client gives up immediately rather
than replaying a command that may already have committed. The default would have been 16 replays.
The accompanying code comment states the intent accurately.
6.8 node-redis 6 adoption — FIXED, option names verified
All Sentinel options used by normalizeSentinelOptions exist in sentinel/types.d.ts:
sentinelRootNodes (28), maxCommandRediscovers (32), nodeClientOptions (38),
sentinelClientOptions (44), passthroughClientErrorEvents (72). No typos, no invented options.
(Cosmetic: RedisSentinelNodeClientOptions is Omit<…, keyof SentinelCommander>, and
SentinelCommander includes RESP, so setting RESP inside nodeClientOptions is type-redundant.
Harmless at runtime because the top-level RESP is what propagates.)
6.9 Cluster readiness and hash slots — FIXED
Key builders use explicit hash tags:
_docBase = `${prefix}{editor:${b64(tenant)}:${b64(docId)}}:`
_statBase = `${prefix}{stat:${b64(tenant)}}:`
// global indexes: `${prefix}{editor:index}:documents` and friendsEvery multi-key Lua script stays within one tag:
| Script | Keys | Tag |
|---|---|---|
ADD_PRESENCE / UPDATE_PRESENCE / GET_PRESENCE / REMOVE_PRESENCE |
presence set/hash/version | {editor:T:D} |
CLEAN_DOCUMENT |
9 per-document keys | {editor:T:D} |
POP_EXPIRED / ACK_EXPIRED |
index, lease, claims | {editor:index} |
ADD_UNIQUE_USER / GET_UNIQUE_USERS / ADD_MONTH_USER |
expiry + info | {stat:T} |
SET_SHARD_COUNT / INCR_SHARD_COUNT / GET_SHARD_COUNT |
count + updated | {stat:T} |
base64url encoding of tenant and docId prevents a hostile or merely awkward identifier from
injecting {/} and breaking co-location. contract.tests.js asserts both the co-location and the
encoding.
Cluster EVAL routing is correct: base.js:532 picks normalized[3], which is KEYS[1] in
EVAL script numkeys key1 …. Every _eval call site passes at least one key, so the zero-key
mis-routing case does not arise.
6.10 cleanDocumentOnExit unconditional-delete concern — ADDRESSED
Now conditional on the presence hash being empty. See Major 2
for the remaining documentation gap — the behaviour itself is the safer choice.
6.11 Redis failure policy phase one — IMPLEMENTED AS DESCRIBED
// base.js:678-706
_checkAndLock -> catch -> return false // fail closed
_checkAndUnlock -> catch -> return c_oAscUnlockRes.Locked // fail closed// editorStat.js:222-230
lockNotification -> catch -> return false // fail closedReaders (getPresence, getLocks, getMessages, getForceSave,
getDocumentPresenceExpired, getForceSaveTimer) propagate rather than returning empty values, and
failure.tests.js asserts each of these. cleanDocumentOnExit aborts before the index operations if
the first eval throws, asserted by evalCalls === 1.
6.12 Redis replication failover limitation — ACCURATELY DOCUMENTED
Readme.md states:
Redis replication failover is not currently lock-safe. If Redis fails immediately after granting a
lock, the promoted replica may not have received that lock, allowing another client to acquire it.
[…] Sentinel support, command retries, andWAITdo not remove this limitation.
This is correct, and correct for the right reasons:
client SET lock NX PX -> primary accepts, replies OK
(replication is asynchronous — the replica may not have it)
primary fails
Sentinel/Cluster promotes the replica
second client SET lock NX PX -> succeeds on the new primary
=> the same lock is held twice
It applies to standalone-with-replica, Sentinel, and Cluster alike, because all three use
asynchronous replication. WAIT does not close it — WAIT is a durability barrier, not a consensus
primitive, and it cannot prevent a failover that promotes a replica which acked fewer writes. The
documentation claims no stronger guarantee than the code provides. This is a documented scope
limitation, not a blocker; it does warrant an operational warning in deployment docs, which is a
reasonable follow-up rather than a merge gate.
6.13 pkg tracing of the six-file module — MECHANISM CORRECT
See §2. Manifest-driven pkg . from
DocService, pkg.scripts entry unchanged, static require chain through all six files.
6.14 Obsolete node-redis 4.7.0 concern — RESOLVED
DocService/package.json now pins "redis": "6.2.1"; ioredis is removed. 3DPARTY.md is updated
to match. No remaining require('ioredis') anywhere.
7. Minor issues
| # | Issue | Location | Detail |
|---|---|---|---|
| m1 | CI path filters incomplete | redisEditorDataTests.yml:8-25 |
Omits DocService/sources/editorDataRedis.js (the pkg shim) and DocService/sources/gc.js. gc.js implements the ACK half of the GC protocol; a gc.js-only change would skip these tests entirely. |
| m2 | Asymmetric topology validation | base.js:211-219 |
Sentinel throws on incomplete config (requires optionsSentinel.name / .sentinelRootNodes), but optionsCluster missing rootNodes silently falls back to standalone. The brief asks for loud failure on ambiguous config; Cluster does not comply. |
| m3 | Reduced GC batch size | base.js:73 |
POP_EXPIRED_BATCH_SIZE = 100. A large expired backlog drains at 100 per expire.documentsCron tick. |
| m4 | Non-atomic index update on all topologies | editorData.js:88-92, 105 |
The presence/index split is required for Cluster but applied unconditionally, so standalone also gets the window. Bounded and self-healing on the next presence write; the REMOVE_DOCUMENT_INDEX_SCRIPT score guard prevents deleting a fresher entry. |
| m5 | One connection per instance | base.js:630-632 |
EditorCommon constructs its own RedisConnection. DocsCoServer ×2 + routes/info.js:22 + notificationService.js:38 (+ proxy) ⇒ 4–5 connections per replica. close() only closes its own. |
| m6 | command() client race |
base.js:522-538 |
Reads this.client after await this.connect(); a concurrent _abortClient() triggered by another command's timeout nulls it, producing a TypeError rather than a typed error. Narrow, and fail-closed in effect. |
| m7 | Quota reads degrade to zeros | routes/info.js:227-246 |
getEditorConnectionsCount, getLiveViewerConnectionsCount, getPresenceUniqueUser, getPresenceUniqueUsersOfMonth feed output.quota. A Redis error lands in the outer catch at line 271, leaving quota fields at their defaults — under-reporting rather than failing visibly. |
| m8 | Stale comment | canvasservice.js:1316 |
"clean redis (redisKeyPresenceSet and redisKeyPresenceHash removed with last element)" no longer describes what cleanDocumentOnExit does. |
| m9 | ADD_MONTH_USER on a TTL-less key |
scripts.js:261-269 |
PTTL returns -1 for an existing key with no TTL, which re-triggers the ZADD + PEXPIRE branch. Only reachable if the TTL were externally cleared. |
8. Full API audit
EditorData — 28 methods. "Tested?" = covered by an executable assertion somewhere in tests/redis/.
| Method | Atomic? | Idempotent? | Failure result | Retry safe? | Multi-replica safe? | Tested? |
|---|---|---|---|---|---|---|
addPresence |
Per-key script; index ZADD separate | Yes | throws | SAFE | Yes (bounded index window) | Yes (process, presence) |
updatePresence |
Same | Yes | throws | SAFE | Yes | Yes (presence) |
removePresence |
Yes (script) | Yes | throws | SAFE | Yes | Yes (presence) |
getPresence |
Yes (script) | Read | throws | SAFE | Yes | Yes (process, presence) |
getDocumentPresenceExpired |
Yes (script) | Claim | throws | SAFE (lease) | Yes | Yes (expiration ×5) |
removePresenceDocument |
2 scripts, version-guarded | Yes | throws | SAFE | Yes (score CAS) | Yes (process, presence) |
lockSave |
Yes (script) | Yes (same token) | false |
SAFE | Yes† | Yes (process, failure) |
unlockSave |
Yes (script) | Yes | Locked |
SAFE | Yes | Yes (process, failure) |
lockAuth |
Yes (script) | Yes | false |
SAFE | Yes† | Yes (failure) |
unlockAuth |
Yes (script) | Yes | Locked |
SAFE | Yes | Yes (failure) |
addLocks |
Yes (script) | Yes (HSET) |
throws | SAFE | Yes | Yes (process) |
addLocksNX |
Yes (script) | Yes (HEXISTS+HSET) |
throws | SAFE | Yes | Yes (process, edge) |
removeLocks |
Yes (script) | Yes (HDEL) |
throws | SAFE | Yes | Yes (process) |
removeAllLocks |
Yes (DEL) |
Yes | throws | SAFE | Yes | No |
getLocks |
Yes | Read | throws | SAFE | Yes | Yes (process, failure) |
addMessage |
Yes (script) | No (RPUSH) |
throws | UNSAFE | Yes | Yes (process) |
removeMessages |
Yes (DEL) |
Yes | throws | SAFE | Yes | No |
getMessages |
Yes | Read | throws | SAFE | Yes | Yes (process, failure) |
setSaved |
Yes (SET EX) |
Yes | throws | SAFE | Last-writer-wins | Yes (process) |
getdelSaved |
Yes (script) | No (consumes) | throws | UNSAFE | Yes | Yes (process) — happy path only |
setForceSave |
Yes (script) | Yes (full overwrite) | throws | SAFE | Last-writer-wins | Yes (process, edge) |
getForceSave |
Yes (HMGET) |
Read | throws | SAFE | Yes | Yes (process, failure) |
checkAndStartForceSave |
Yes (script) | Yes (started guard) |
throws | SAFE | Yes (one winner) | Yes (process ×2 procs) |
checkAndSetForceSave |
Yes (script) | Yes (time/index CAS) | throws | SAFE | Yes (stale rejected) | Yes (process, edge) |
removeForceSave |
Yes (DEL) |
Yes | throws | SAFE | Last-writer-wins | No |
cleanDocumentOnExit |
Script + 2 follow-ups | Yes | throws (aborts early) | SAFE | Yes (presence-guarded) | Yes (edge, presence) |
addForceSaveTimerNX |
Yes (ZADD NX) |
Yes | throws | SAFE | Yes (first write wins) | Yes (process) |
getForceSaveTimer |
Yes (script) | Claim | throws | SAFE (lease) | Yes | Yes (expiration ×5) |
† Subject to the documented replication-failover limitation (§6.12).
EditorStat — 28 methods. Rows collapse generated families (e.g. the three
set*ConnectionsCountByShard variants share one row).
| Method | Atomic? | Idempotent? | Failure result | Retry safe? | Multi-replica safe? | Tested? |
|---|---|---|---|---|---|---|
addPresenceUniqueUser / …ViewUser |
Yes (script) | Yes | throws | SAFE | Yes | Yes (process, edge) |
getPresenceUniqueUser / …ViewUser |
Yes (script) | Read+evict | throws | SAFE | Yes | Yes (process) |
addPresenceUniqueUsersOfMonth / …View… |
Yes (script) | Yes | throws | SAFE | Yes | Yes (process, edge) |
getPresenceUniqueUsersOfMonth / …View… |
No (ZREM + ZRANGE + MULTI) | Read | throws | SAFE | Yes | Yes (process, edge) |
setEditorConnections |
Yes (script) | Yes (unique member) | throws | SAFE | Yes (per-shard sampleId) |
Yes (edge) |
getEditorConnections |
Yes | Read | throws | SAFE | Yes | Yes (edge) |
set{Editor,Viewer,LiveViewer}ConnectionsCountByShard |
Yes (script) | Yes (HSET) |
throws | SAFE | Yes | Yes (shard-expiration) |
incr{…}ConnectionsCountByShard |
Yes (script) | No (HINCRBY) |
throws | UNSAFE | Yes | Yes (shard-expiration) |
get{…}ConnectionsCount |
Yes (script) | Read+evict | throws | SAFE | Yes | Yes (shard-expiration) |
addShutdown |
Yes (SADD) |
Yes | throws | SAFE | Yes | No |
removeShutdown |
Yes (SREM) |
Yes | throws | SAFE | Yes | No |
getShutdownCount |
Yes (SCARD) |
Read | throws | SAFE | Yes | No |
cleanupShutdown |
Yes (DEL) |
Yes | throws | SAFE | Yes | No |
setLicense |
Yes (HSET) |
Yes | throws | SAFE | Last-writer-wins | No |
getLicense |
Yes (HGET) |
Read | throws | SAFE | Yes | No |
removeLicense |
Yes (HDEL) |
Yes | throws | SAFE | Yes | No |
lockNotification |
Yes (SET NX PX) |
Yes | false |
SAFE | Yes | Yes (process, failure) |
deleteKey |
Yes (DEL) |
Yes | throws | SAFE | Yes | No — see Major 4 |
Do any of these statistics actually matter?
The brief asks not to dismiss anything as "only statistics". They are not:
- Licensing / quota —
routes/info.js:227-246feedsoutput.quota.edit.connectionsCount,
output.quota.view.connectionsCount,usersCount.unique,usersCount.anonymousand
quota.byMonthfromgetPresenceUniqueUser,getEditorConnectionsCount,
getLiveViewerConnectionsCountandgetPresenceUniqueUsersOfMonth. These are the licence-facing
numbers. - Administrative behaviour —
getShutdownCountgates the shutdown drain loop
(shutdown.js:72-74). incr*ConnectionsCountByShardis not retry-safe, but self-heals:DocsCoServer.jsoverwrites
each shard's authoritative count withset*ConnectionsCountByShardon everyexpireDoctick, so
HINCRBYdrift cannot accumulate.INCR_SHARD_COUNT_SCRIPTadditionally clamps negatives to zero.- Stale-shard decay works:
SET/INCRrefresh both the count hash and the shard heartbeat ZSET
withexpire.shard(300 s, new in this PR), andGET_SHARD_COUNT_SCRIPTevicts shards whose
heartbeat is older than that window before summing.
9. Failure policy matrix
| Method | Current policy | Correct? | Retry safe? | Remaining concern |
|---|---|---|---|---|
lockSave / lockAuth |
false on error |
Yes | SAFE | A lost reply leaves the lock held until TTL — liveness, not safety |
unlockSave / unlockAuth |
Locked on error |
Yes | SAFE | — |
lockNotification |
false on error |
Yes | SAFE | — |
getPresence / getLocks / getMessages / getForceSave |
propagate | Yes | SAFE | — |
getDocumentPresenceExpired / getForceSaveTimer |
propagate | Yes | SAFE | — |
addPresence / updatePresence |
propagate | Yes | SAFE | Index ZADD non-atomic; bounded, self-heals |
removePresence / removePresenceDocument |
propagate | Yes | SAFE | — |
addLocks / addLocksNX / removeLocks / removeAllLocks |
propagate | Yes | SAFE | — |
addMessage |
propagate | Tolerable | UNSAFE (RPUSH) |
A retry duplicates a message; no caller retries today |
setSaved |
propagate | Yes | SAFE | — |
getdelSaved |
propagate | No | UNSAFE | Major 1 — needs an "unknown" state |
setForceSave |
propagate | Yes | SAFE | — |
checkAndStartForceSave / checkAndSetForceSave |
propagate | Yes | SAFE (CAS) | — |
removeForceSave |
propagate | Yes | SAFE | Stale remove can drop a newer record (same as memory) |
cleanDocumentOnExit |
propagate, aborts before index ops | Yes | SAFE | Silent no-op divergence (Major 2) |
set*ConnectionsCountByShard |
propagate | Yes | SAFE | — |
incr*ConnectionsCountByShard |
propagate | Tolerable | UNSAFE | Self-heals via periodic set* overwrite |
addShutdown / removeShutdown / cleanupShutdown |
propagate | Yes | SAFE | — |
setLicense / removeLicense |
propagate | Yes | SAFE | — |
deleteKey |
propagate | Questionable | SAFE | Guessed key format (Major 4) |
Is the partial failure policy internally safe? Broadly yes. The two decisions that can be
irreversible are lock acquisition and destructive cleanup, and both fail closed: locks deny, and
cleanDocumentOnExit throws before reaching the index/timer removals. The one place where a Redis
failure can still cause a decision on false information is getdelSaved, because null means both
"nothing recorded" and "we lost the answer".
10. Lost-response / retry classification
Applying the brief's T0–T4 scenario to every mutating method:
SAFE TO RETRY — lockSave, lockAuth (same fencing token re-acquires), unlockSave,
unlockAuth, lockNotification, addPresence, updatePresence, removePresence,
removePresenceDocument, addLocks, addLocksNX, removeLocks, removeAllLocks,
removeMessages, setSaved, setForceSave, removeForceSave, addForceSaveTimerNX,
cleanDocumentOnExit, set*ConnectionsCountByShard, setEditorConnections, addShutdown,
removeShutdown, cleanupShutdown, setLicense, removeLicense, deleteKey, the unique-user and
monthly-user writers.
UNSAFE TO RETRY
addMessage—RPUSHappends; a retry duplicates. No caller retries today, so this is latent.incr*ConnectionsCountByShard—HINCRBYaccumulates; mitigated by the periodic authoritative
set*overwrite.
REQUIRES CLAIM/ACK PROTOCOL — getDocumentPresenceExpired, getForceSaveTimer.
Already implemented correctly (§6.3).
REQUIRES READ-BACK or an explicit "unknown" state — getdelSaved. Not implemented (Major 1).
REQUIRES VERSION — removePresenceDocument and cleanDocumentOnExit's index removal.
Already implemented via the presence-version CAS in REMOVE_DOCUMENT_INDEX_SCRIPT.
The general point the brief makes holds here and the implementation mostly respects it: "Redis
command is atomic" does not mean the application operation is retry-safe. getdelSaved is the one
place where that gap is still open.
11. Key and schema review
| Key | Scope | Hash tag | TTL | Notes |
|---|---|---|---|---|
{editor:T:D}:presence:set |
document | doc | expire.presence |
ZSET userId → expireAt |
{editor:T:D}:presence:hash |
document | doc | expire.presence |
HASH userId → info |
{editor:T:D}:presence:version |
document | doc | expire.presence |
CAS token for index removal |
{editor:T:D}:savelock / :lockdocument |
document | doc | ttl arg (PX) |
fencing token |
{editor:T:D}:locks |
document | doc | expire.locks |
HASH lockId → JSON |
{editor:T:D}:message |
document | doc | expire.message |
LIST |
{editor:T:D}:saved |
document | doc | expire.saved |
STRING |
{editor:T:D}:forcesave |
document | doc | expire.forcesave |
HASH, 10 fields |
{editor:index}:documents |
global | index | none | pruned by GC |
{editor:index}:forcesavetimer |
global | index | none | pruned by GC |
{editor:index}:*:expired:lease |
global | index | none | GC lease ZSET |
{editor:index}:*:expired:claims |
global | index | none | GC claim HASH |
{stat:T}:presence:unique:{edit,view}:{expiry,info} |
tenant | stat | none | — |
{stat:T}:presence:month:{edit,view}:index / :<b64period> |
tenant | stat | expire.monthUniqueUsers |
fixed at creation |
{stat:T}:editorconnections |
tenant | stat | none | trimmed by score |
{stat:T}:connections:<type>:{count,updated} |
tenant | stat | expire.shard (300 s) |
new in this PR |
{stat:T}:notification:<b64type> |
tenant | stat | ttl arg | SET NX PX |
| caller-supplied (shutdown, licence) | caller | none | none | used verbatim, correctly un-prefixed |
Assessment against the brief's checklist:
- Scope — correct; documents are tenant+doc scoped, stats tenant scoped, indexes global.
- Hash slots — correct; no multi-key script crosses a tag (§6.9).
- Co-location — required groups share a tag.
- TTLs — every per-document key carries one, so an
allkeys-*eviction policy is not the only
thing keeping the keyspace bounded. The global indexes deliberately have none and rely on GC. - Can stale state overwrite fresh state? — guarded where it matters:
REMOVE_DOCUMENT_INDEX_SCRIPTcompares the presence version,SET_FORCE_SAVE_SCRIPTcompares
time/index,ACK_EXPIRED_SCRIPTcompares the claim token,addForceSaveTimerNXusesZADD NX.
The remaining unguarded case is the plainZREM forceSaveTimerKeyateditorData.js:337, which
could drop a timer re-added between the clean script and the ZREM. Narrow. - Can cleanup delete newer state? — no, because of the presence-empty guard plus the version CAS.
- Cross-tenant interference — no; tenant is base64url-encoded into every tag.
12. Test quality audit
What genuinely proves an invariant
| Invariant | Test that proves it |
|---|---|
| Stale ACK cannot remove a newer claim | expiration.tests.js — 'does not let a stale document-presence acknowledgement remove a retry' (and the force-save twin) |
| A crashed worker's unprocessed items come back | 'recovers only the unacknowledged document-presence item after a partial batch' |
| A lost reply does not lose expired work | 'recovers a document-presence entry after a discarded response', '…after a client timeout' |
| Exactly one replica starts a force-save | process.tests.js:444-449, across two real OS processes |
| A stale force-save update cannot overwrite | process.tests.js:450-460 |
| Cleanup does not delete a live document | presence.tests.js — 'document cleanup does not delete presence while a user remains' |
| The surviving replica stays usable after SIGKILL | process.tests.js:577 |
| Redis and memory agree where they should | process.tests.js:551 differential trace |
| Cluster co-location holds | contract.tests.js — 'isolates document keys and co-locates one document for Redis Cluster' |
| Client resets after a command timeout | editorDataRedis.tests.js — 'resets the client after a command timeout' |
That is a genuinely good set. The claim/lease/ACK protocol in particular is tested against exactly
the adversarial sequences that matter.
Where the confidence is false
| Pattern | Where | Consequence |
|---|---|---|
| Mocks Redis instead of failing it | failure.tests.js — patches _eval / _command |
Only tests "failed before mutation"; never a real network fault, kill, or pause |
| Tests only successful commands | most of edge, presence, contract |
— |
| Does not simulate lost responses outside GC | everywhere except expiration.tests.js and one addPresence case |
getdelSaved, setSaved, addMessage, force-save all unverified under commit-then-lost-reply, despite holdAfterCommand making it cheap |
| Mocks failover | nothing tests failover at all | The documented lock limitation is untested, as is Sentinel promotion |
| Single process | all suites except process.tests.js |
— |
| Skipped under Cluster | process.tests.js:547-548 (describeStandalone) |
No cross-process verification on Cluster |
Remaining test gaps by layer
- Unit — none material.
- Integration —
cleanDocumentOnExitconditional semantics not pinned against
editorDataMemory;removeAllLocks,removeMessages,removeForceSave, shutdown tracking and
licence storage have no assertions at all. - Multi-process — only two tests; skipped under Cluster; oracle coverage gaps listed in §6.5.
- Failure injection — no test kills or pauses a real Redis; no lost-reply coverage outside GC.
- Cluster — job exists, last run failed on infrastructure, has not re-run; process tests skipped.
- Sentinel — no integration coverage of any kind (Major 3).
- Package / E2E — correctly designed, never executed, and pins
optionsCluster/optionsSentinel
to{}so it cannot catch entrypoint config drift.
13. Requirements compliance
Against issue #267 as the specification.
| Requirement | Status | Evidence |
|---|---|---|
editorDataRedis module exists and resolves |
Fully implemented | DocService/sources/editorDataRedis.js + six-file module |
| Module traced into the pkg binary | Implemented, unverified | pkg.scripts entry + static require chain; no executed build |
Packaged DocService starts with editorDataStorage: "editorDataRedis" |
Unverified | pkg-e2e.sh exists and is well-designed; never ran (Blocker 1) |
Works from minimal redis: {host, port} in local.json |
Fully implemented | default.json supplies options/optionsCluster/optionsSentinel; hasNodeCluster/hasNodeSentinel both false ⇒ standalone |
| Docker and native builds consistent | Fully implemented | Both use pkg . from DocService |
| Session state survives DocService restart | Fully implemented | Redis-backed presence, locks, messages, saved, force-save |
| Compatible with existing deployment config | Contradicted for Sentinel | iooptions vs optionsSentinel (Blocker 2) |
editorDataStorage default unchanged |
Fully implemented | still editorDataMemory |
Ambiguous / not specified by #267 (decided by the PR, worth an explicit maintainer sign-off):
- Whether Cluster and Sentinel are in scope at all — #267 says nothing about them.
- Whether
deleteKeyshould be implemented or remain a no-op. - Whether
cleanDocumentOnExitmay become conditional. - What the failure policy should be for the ~14 methods still outside phase one.
14. Security and isolation
| Area | Assessment |
|---|---|
| Credentials | Read from config (options.user/password, Sentinel sentinelClientOptions). Never logged — _createClient's debug line prints only connector, cluster flag, host, port, database. |
| TLS | Passed through via socket options; not independently validated, and untested for Sentinel. |
| Configuration logging | Clean; no credential leakage found. |
| Key construction | Good. encodePart base64url-encodes tenant and docId, so neither {/} nor : from a hostile identifier can break hash tags or forge another tenant's key. contract.tests.js asserts this, including a tenant:{}:/世界 case. |
| Tenant isolation | Enforced structurally by the tag; edge.tests.js asserts save locks are isolated across tenants with identical doc IDs. |
| Malicious key values | Lock values and messages are JSON-encoded on write and jsonDecoded with a fallback on read, so malformed data degrades to null rather than throwing. |
| Cross-database operations | The EditorStat(database) proxy path opens a separate connection on the configured logical DB, and deleteKey issues an unprefixed DEL there — see Major 4. |
| Prefix | cfgRedisPrefix is not encoded, but it is administrator-controlled config, not user input. |
15. Final assessment
Needs significant changes
To be clear about what that means: the implementation itself is close. The remaining work is
verification and a config bridge, not a rewrite. The distributed-systems design — claim/lease/ACK,
hash-tag co-location, opaque force-save payloads, fail-closed locks, per-claim tokens — is sound and,
where CI actually runs, well tested.
Required before merge
- Approve and run both workflows on
ec7900f5; get them green, especially the packaged smoke test.
This is non-negotiable for a packaging bug. - Resolve the
iooptions→optionsSentinelmismatch: bridge it in code, land the companion
DocumentServer change, or drop the Sentinel path.
Should be decided before merge
getdelSavedretry semantics — introduce a distinct "unknown" result.- Document the
cleanDocumentOnExitdivergence and fix the stale comment atcanvasservice.js:1316. - Decide whether
deleteKeyshould really delete.
Cheap follow-ups
- Add a Sentinel CI job.
- Extend
holdAfterCommandtogetdelSaved,setSavedandaddMessage. - Add
DocService/sources/gc.jsand the pkg shim to the workflow path filters. - Add an operational warning about the replication-failover lock limitation to deployment docs.
Appendix A — CI evidence
PR #46 HEAD = ec7900f516271e4e42a1fb029f09a05d06d747fc
base = main (merge-base abfa2946fcc7b6f31df497f73d7e48ca6dbef16b), 18 commits, 32 files
$ gh run list --branch feature/redis-editor-data-storage
2026-09-23T18:13 5e3043c1 failure Redis editor data tests
2026-09-23T18:13 5e3043c1 success E2E (overlay on nightly)
2026-09-23T17:51 ee229022 action_required E2E (overlay on nightly)
2026-09-23T17:51 ee229022 action_required Redis editor data tests
2026-09-23T17:43 913378ad action_required Redis editor data tests
2026-09-23T17:43 913378ad action_required E2E (overlay on nightly)
2026-09-23T16:22 1f9f803d failure Redis editor data tests
2026-09-23T16:22 1f9f803d success E2E (overlay on nightly)
2026-09-21T14:36 cf0d7200 action_required (both)
2026-09-21T14:15 e1a4ad50 action_required (both)
2026-09-18T15:37 4466ee4d action_required (both)
$ gh run list --commit ec7900f5...
Redis editor data tests -> action_required
E2E (overlay on nightly) -> action_required
$ gh run view 35901026235 --json jobs
Redis Cluster editor data tests: failure
Redis editor data tests: success
$ gh api .../jobs/107501320437/logs | tail
/home/runner/work/_temp/....sh: line 13: redis-cli: command not found (×30)
##[error]Process completed with exit code 1.
$ git ls-tree 5e3043c1 -r --name-only | grep -i pkg
tests/redis/editorDataRedis.pkg-smoke.js # pkg-e2e.sh absent
Only the DCO check is currently passing on the PR.
Appendix B — node-redis 6.2.1 verification
Downloaded redis@6.2.1 and @redis/client@6.2.1 and read the source rather than trusting commit
messages.
dist/lib/cluster/cluster-slots.js:465 options.RESP = this.#options.RESP;
dist/lib/cluster/cluster-slots.js:550 RESP: this.#options.RESP,
=> top-level cluster RESP propagates to every node client.
=> normalizeClusterOptions setting options.RESP = 2 and deleting defaults.RESP is CORRECT.
dist/lib/sentinel/index.js:600-601 // first take the globally set RESP
RESP: this.#RESP,
=> Sentinel likewise propagates.
dist/lib/sentinel/index.js:561 this.#maxCommandRediscovers = options.maxCommandRediscovers ?? 16;
dist/lib/sentinel/index.js:720 if (++count > this.#maxCommandRediscovers) { ... }
=> maxCommandRediscovers: 0 gives up on the first attempt; no replay of a committed command. CORRECT.
dist/lib/sentinel/types.d.ts:28 sentinelRootNodes: Array<RedisNode>;
dist/lib/sentinel/types.d.ts:32 maxCommandRediscovers?: number;
dist/lib/sentinel/types.d.ts:38 nodeClientOptions?: RedisSentinelNodeClientOptions<...>;
dist/lib/sentinel/types.d.ts:44 sentinelClientOptions?: RedisSentinelNodeClientOptions<...>;
dist/lib/sentinel/types.d.ts:72 passthroughClientErrorEvents?: boolean;
=> all option names used by normalizeSentinelOptions are real. No typos.
redis@6.2.1 dependencies: @redis/bloom, @redis/client, @redis/json, @redis/search, @redis/time-series
=> the Redis-Stack sub-packages are still pulled in, which inflates the pkg binary.
Not a correctness issue against a plain Redis/Valkey target; worth noting for binary size.
|
This PR is moving at lightning speed 😊 I started a deep review (at head I went looking from the angle of how we would want to run it in a multi-replica setup and general robustness, focusing only on things that surfaced in the analysis and were not already raised:
Hope these insights can help, and that the entrypoint config question you raised can be clarified soon, @nvanlaerebeke. Really nice progress overall 🤙 |
|
@j-base64 , the force-save was a problem, I created some tests that verified that there were issues before I applied a fix |
|
@j-base64 , I went over your points first as the changes were not too big. Thanks for the thorough review and for cross-checking everything against the previous feedback 😊 I went through each point and pushed follow-up changes:
Thanks again for the careful review. |
|
Took me a bit to get it into a state I liked, I've opened up a PR (Euro-Office/DocumentServer#386) on the DocumentServer project with the changes for the sentinel configuration: Euro-Office/DocumentServer#386 It also includes easy to run commands to validate the setups for both Redis and Valkey in:
So at the time of writing I've been able to test each topology from a locally build image both with redis and valkey. I'll go over the remaining items next |
|
Thanks for the connection-layer work and the failure-handling tests. One regression in the connection layer needs fixing: A single failed connect disables Redis for the life of the process. At Fix: keep Details
test('a failed connect() does not prevent the next command() from connecting', async () => {
const connection = new RedisConnection();
connection.connector = 'redis';
connection.cluster = false;
connection.client = fakeClient({
isOpen: false,
isReady: false,
connect: async () => {
throw new Error('Connection timeout');
}
});
await assert.rejects(connection.connect(), /Connection timeout/);
try {
assert.equal(await connection.command(['PING']), 'PONG');
} finally {
await connection.close();
}
});
|
|
Proposal: land #46 as a stack of smaller PRs rather than one. It is currently +7,533 lines across 56 files, and at that size a problem in one area holds up everything else. Feedback welcome, from the author and from anyone else working on this code, before anything gets cut. A stack keeps one branch to test: the top one carries everything below it, so anyone building #46 today would build that branch instead. Rough shape, bottom first:
Open questions:
Details
|
|
I re-checked Not raised yet1. Regression on the default in-memory backend: every final save is treated as failed (reproduced) 2. SIGTERM, SIGINT and 3. Sentinel mode runs commands one at a time (reproduced) 4. Lost presence entries are never rebuilt, so live editors can vanish
The next editor who leaves then triggers the final save and 5. The Helm chart can't configure Sentinel for this PR 6. Two additions to @moodyjmz's startup-latch point
Should also be fixed (medium)
Low / cleanup
|
Rebuild missing presence entries during valid heartbeats, guard removals by connection ID, and keep the document presence index synchronized across replicas, TTL gaps, and cleanup. Add Redis-backed regression coverage for recovery, reconnect races, replica coordination, and live-user cleanup/GC behavior. Assisted-by: Codex:GPT-5 Signed-off-by: Nico Van Laerebeke <1520618+nvanlaerebeke@users.noreply.github.com>
Add dedicated TTLs and legacy migration for saved-state claims, make acknowledgements idempotent, and prevent stale workers from deleting newer saved state. Expand Redis and callback regression coverage. Assisted-by: Codex:GPT-5 Signed-off-by: Nico Van Laerebeke <1520618+nvanlaerebeke@users.noreply.github.com>
- retry initial Sentinel discovery without replaying commands - support concurrent Sentinel operations with reserved clients - recover existing clients after Sentinel failover - validate Redis and Sentinel credentials and topology settings - expand authenticated, unauthenticated, ACL, Cluster, and failover coverage - document Sentinel connection and authentication behavior Assisted-by: Codex:GPT-5 Signed-off-by: Nico Van Laerebeke <1520618+nvanlaerebeke@users.noreply.github.com>
Serialize editor-data operations so a timed-out command can safely abort its physical Redis client without poisoning unrelated operations. Document the tradeoff: a slow command can temporarily hold up other editor-data work, and Cluster batches are no longer fully parallel. A pool of Redis clients could reduce that impact, but would be a much larger and riskier change across the supported topologies. Assisted-by: Codex:GPT-5 Signed-off-by: Nico Van Laerebeke <1520618+nvanlaerebeke@users.noreply.github.com>
Process expiry claims independently so failed documents do not block successful items, while preserving Redis lease-based retries. Validate document expiry scheduling against presence and shard TTLs with scheduling headroom and cover the GC entry point with regression tests. Assisted-by: Codex:GPT-5 Signed-off-by: Nico Van Laerebeke <1520618+nvanlaerebeke@users.noreply.github.com>
Preserve the duplicate-start guard while returning NoError with inProgress for Form and Internal force-save callers. Keep normal editor failures and Redis backend errors distinguishable. Add regression coverage for normal, active, concurrent, memory, and Redis failure cases. Assisted-by: Codex:GPT-5 Signed-off-by: Nico Van Laerebeke <1520618+nvanlaerebeke@users.noreply.github.com>
- broaden Redis workflow path filters - run Redis tests in the unit workflow - strengthen packaged Redis smoke coverage - update pkg CLI options for current packaging support Assisted-by: Codex:GPT-5 Signed-off-by: Nico Van Laerebeke <1520618+nvanlaerebeke@users.noreply.github.com>
Add bounded expiration continuation across Redis shards so document-presence and force-save queues continue draining across GC passes without unbounded Redis work. Preserve parallel force-save processing with isolated per-item contexts, handle resolved and thrown failures without acknowledging unsuccessful saves, recover partial shard failures, and expose/log shard failure metadata. Extend expiration, GC scheduling, retry, lease recovery, cluster-routing, and concurrent force-save tests. Assisted-by: Codex:GPT-5 Signed-off-by: Nico Van Laerebeke <1520618+nvanlaerebeke@users.noreply.github.com>
Prevent replica clock skew from expiring live document presence or triggering premature document cleanup. Use Redis TIME in presence Lua scripts and document GC while preserving connection protections, leases, and Cluster key routing. Add regression coverage for clock skew, TTL renewal, expiry, GC, and standalone/Cluster Redis execution. Assisted-by: Codex:GPT-5 Signed-off-by: Nico Van Laerebeke <1520618+nvanlaerebeke@users.noreply.github.com>
Retain a leased resolved marker so duplicate acknowledgements succeed without masking unknown, expired, mismatched, or malformed claims. Preserve claim recovery and protect newer tasks from stale acknowledgements. Add Redis and callback regression coverage for duplicate delivery, concurrent acknowledgements, expiry, malformed claims, and cleanup claim propagation. Assisted-by: Codex:GPT-5 Signed-off-by: Nico Van Laerebeke <1520618+nvanlaerebeke@users.noreply.github.com>
Split Redis scripts, data/stat stores, configuration, and connection handling into focused modules while preserving public APIs and behavior. Assisted-by: Codex:GPT-5 Signed-off-by: Nico Van Laerebeke <1520618+nvanlaerebeke@users.noreply.github.com>
Reject Sentinel commands before initial readiness, support CI-specific Cluster container prefixes, and document local Redis topology test setup. Assisted-by: Codex:GPT-5 Signed-off-by: Nico Van Laerebeke <1520618+nvanlaerebeke@users.noreply.github.com>
Assisted-by: Codex:GPT-5 Signed-off-by: Nico Van Laerebeke <1520618+nvanlaerebeke@users.noreply.github.com>
Preserve active saved-state claims, enforce replica-safe cleanup results, and gate terminal side effects on confirmed cleanup. Add end-to-end disconnect coverage and Redis test readiness checks. Assisted-by: Codex:GPT-5 Signed-off-by: Nico Van Laerebeke <1520618+nvanlaerebeke@users.noreply.github.com>
Add bounded Redis startup retries, cancellation, schema failure handling, and readiness-gated HTTP server startup. Cover recovery, exhaustion, shutdown, overlapping installs, and invalid retry configuration with regression tests. Assisted-by: Codex:GPT-5 Signed-off-by: Nico Van Laerebeke <1520618+nvanlaerebeke@users.noreply.github.com>
Document the saved:claim key lifecycle and configured TTLs, and add the standard SPDX header to the Redis compatibility shim. Assisted-by: Codex:GPT-5 Signed-off-by: Nico Van Laerebeke <1520618+nvanlaerebeke@users.noreply.github.com>
Remove redundant Redis option handling and document the @redis/* packages pulled in by node-redis 6.2.1. Assisted-by: Codex:GPT-5 Signed-off-by: Nico Van Laerebeke <1520618+nvanlaerebeke@users.noreply.github.com>
Share Redis test setup across process and topology suites, and reuse expiration test setup while preserving coverage for both queue types. Assisted-by: Codex:GPT-5 Signed-off-by: Nico Van Laerebeke <1520618+nvanlaerebeke@users.noreply.github.com>
Assisted-by: Codex:GPT-5 Signed-off-by: Nico Van Laerebeke <1520618+nvanlaerebeke@users.noreply.github.com>
457e63f to
f62c462
Compare
|
This PR is now in a good state to start splitting up I think. Since 9050fa9 (f9a2149 after rebase), I have:
I also ran the complete local test suite, including all of the procedures documented in I did not test every Redis/Valkey version from the CI matrix, but the topology coverage was completed locally. I have started splitting up the PR. The first split is #52, which fixes configuration lifecycle issues found while running the Redis tests, I've rebased this branch on top of that PR. The next split will extract saved-state claim handling. I'll also have to do Euro-Office/DocumentServer#386, but for that I'll need a PR for the helm chart as well. So related PRs so far:
Most of this PR are tests, so I'll take it one step at a time. |
|
Thanks for the big push here @nvanlaerebeke, awesome work ! that's a lot of ground covered 💪. |
MonaAghili
left a comment
MonaAghili
left a comment
There was a problem hiding this comment.
Thanks @nvanlaerebeke, splitting it this way works for us and fits the stack @moodyjmz proposed. I checked the current head f62c4625 with the split in mind. The points below are what the slices will need for green CI.
1. The E2E job can't pass while the overlay keeps the nightly's default.json (reproduced)
The overlay step replaces only the service binaries in the nightly image (e2e.yml#L102-L123). DocService then reads the nightly's default.json, which has no services.CoAuthoring.expire.shard, and gc.js#L44 reads that key with config.get when the module loads. DocService exits with Configuration property "services.CoAuthoring.expire.shard" is not defined, and supervisor restarts it in a loop. This also happens on the default in-memory backend. With editorDataRedis selected, it fails next on savedClaim (editorDataSettings.js#L25). The new smoke step waits for /healthcheck until it times out, and the Playwright steps after it are skipped. The job log doesn't show the error, because docker logs doesn't include DocService's own log file.
Suggested fix: in the overlay, also copy the branch's Common/config/default.json to /etc/euro-office/documentserver/default.json. The image's local.json still sets the database values. With that change, the container becomes healthy and editorDataRedis.pkg-e2e.sh passes. When the health check gives up, also print "$EO_LOG"/docservice/err.log from the container, which is where the error above shows up. Any slice that adds a key to default.json and reads it with config.get will hit this, so it's easiest to fix the overlay first.
2. All 11 Redis/Valkey jobs fail on one test (reproduced)
DocsCoServer.disconnect.handler.tests.js#L13-L35 mocks socket.io as a virtual module. If an earlier file in the same Jest worker has already loaded DocsCoServer with the real socket.io, the mock isn't applied, and the test fails at io.engine.on with Cannot read properties of undefined (reading 'on'). In CI (--runInBand, empty cache), canvasservice.saved.tests.js and DocsCoServer.forceSave.tests.js run earlier and load it without the mock; either one alone is enough to make the test fail. Locally, Jest runs previously failed files first, so a second run passes and hides the problem.
Suggested fix: mock the path DocService actually resolves, without virtual:
jest.mock('../../DocService/node_modules/socket.io', () => ({
Server: class {
// same body as now
}
}));
With only that change, tests/redis passed 196 of 196, twice, with an empty cache. DocsCoServer.startup.tests.js#L24 uses the same virtual socket.io mock and is worth changing at the same time.
|
Because we're very interested in this work, I ran an independent deep review (at Sharing them in case they're useful, before or after the PR splitting. 📑 FULL REPORT (click to expand): 6 minor points, none blockingCI is currently red at this head, two separate causesNothing to add to @MonaAghili's findings above. Correctness / parity
The differential suiteThe two-process harness (
There are also a few smaller nits I kept out to keep this focused; happy to share them separately if they'd be useful. Thanks again for all the work here, keen to see the split land. Review assisted by Claude/Opus-4-8 |
Update the orchestrated entrypoint to emit the server’s optionsSentinel structure instead of the legacy iooptions format. Also parse Sentinel and Cluster node lists as comma-separated values. Refs: Euro-Office/server#46 Assisted-by: Codex:GPT-5 Signed-off-by: Nico Van Laerebeke <1520618+nvanlaerebeke@users.noreply.github.com>
Document standalone, Sentinel, Cluster, and Valkey setup and verification. Clarify published-image defaults and local-image validation. Update AGENTS.md with the new workflow guidance. Refs: Euro-Office/server#46 Assisted-by: Codex:GPT-5 Signed-off-by: Nico Van Laerebeke <1520618+nvanlaerebeke@users.noreply.github.com>
Update the orchestrated entrypoint to emit the server’s optionsSentinel structure instead of the legacy iooptions format. Also parse Sentinel and Cluster node lists as comma-separated values. Refs: Euro-Office/server#46 Assisted-by: Codex:GPT-5 Signed-off-by: Nico Van Laerebeke <1520618+nvanlaerebeke@users.noreply.github.com>
Document standalone, Sentinel, Cluster, and Valkey setup and verification. Clarify published-image defaults and local-image validation. Update AGENTS.md with the new workflow guidance. Refs: Euro-Office/server#46 Assisted-by: Codex:GPT-5 Signed-off-by: Nico Van Laerebeke <1520618+nvanlaerebeke@users.noreply.github.com>
Thanks! @j-base64, if you'd like to test this, the easiest way would be to deploy it using the Helm chart. Here's the setup I've been using for testing:
This combination has been working well for my tests so far and should provide a good starting point for testing it in your own infrastructure. |
4 participants
Summary
This pull request adds an optional Redis-backed editor-data implementation for the Euro Office Co-Authoring service. It is intended to allow multiple Document Server instances to share the state required for high-availability
deployments.
The implementation covers the shared state used by Co-Authoring, including:
Redis is used for shared editor-data and statistics state.
RabbitMQ remains responsible for the existing pub/sub and task-queue mechanisms.
The backend is disabled by default. It can be selected through the existing configuration:
{ "services": { "CoAuthoring": { "server": { "editorDataStorage": "editorDataRedis", "editorStatStorage": "editorDataRedis" } } } }The Redis connection layer supports standalone Redis, Redis Cluster, and Redis Sentinel, including authenticated, unauthenticated, and ACL configurations.
The same topology support is available for Valkey. Configuration is normalized for the node-redis client, with separate options for Redis nodes and Sentinel nodes.
Implementation details
Cross-document indexes are distributed over fixed shards; related keys use
Redis Cluster hash tags so operations remain slot-compatible.
unbounded single operation.
and force-save updates are made idempotent.
This work depends on Euro-Office/server#52, which adds lifecycle management for the runtime configuration watcher.
Redis shutdown and reconnect paths made the existing watcher/timer cleanup issue observable; without #52, those resources could survive shutdown or cause a reload during teardown and make process/topology tests hang or fail.
One known limitation remains: Redis/Valkey replication failover is not fully lock-safe. If a primary fails immediately after granting a lock, a promoted replica may not yet contain that lock. Sentinel support, retries, and
WAITcannot eliminate that replication window, so deployments should not treat replication failover as a strict lock-safety guarantee.Testing
The test suite now includes:
editorDataRedisthrough the installed configuration and verifies a live Redis connectiontests introduced by the prerequisite in server#52.
The GitHub Actions coverage runs the Redis editor-data suite against:
The regular service unit-test workflow also runs the Redis suite, and the packaged end-to-end workflow runs the Redis smoke test against the packaged server image. Local test instructions and topology setup are documented in
server/tests/redis/README.md.Review notes
The implementation started as an AI-assisted prototype and has since been substantially refactored, reviewed, tested, and hardened through the work described above.
It is submitted for review as a candidate implementation of Redis-backed high availability for
Co-Authoring.