Fix reliable-channel message loss with multiple reliable subscribers

[rudicus]

Hey Dave - Hope all is well. Been using subspace regularly and love it. I ran into an issue with backpressure + multiple reliable subscribers. I was getting drops when one of the reliable subscribers read it, and the other one wasnt able to. Pardon the Claude PR, but this fixes it for me and lets me get 100% reliable playback with back pressure +multiple subscribers.

The bug

A reliable channel loses messages for its slower reliable subscribers the moment it has more than one subscriber:

• A reliable subscriber only holds a slot reference while it is reading a message — between reads it holds nothing.
• kMessageSeenByReliable is set by the first reliable subscriber to read a slot.
• So with two reliable subscribers, the faster one marks a slot seen, and the slower one's unread backlog carries zero references whenever it sits between messages. FindFreeSlotReliable walks past the seen flag, finds zero refs, and reclaims the slot — the slower reliable subscriber silently loses that message (no drop counter fires; the loss is invisible on both ends).

We measured this in production (an autonomy stack replaying sensor data through reliable channels at 64×): a ~4.2k msg/s reliable channel lost 0.05–0.14% of messages whenever it had a second subscriber — whether that second subscriber was reliable or not — and was exactly lossless (1,032,252 / 1,032,252) with a single subscriber. #104's kMessageSeenByReliable fixes the unreliable-co-subscriber case but not the reliable–reliable one.

The fix

The per-subscriber available-slot bitsets already record exactly the needed state — "this slot has not yet been delivered to subscriber X" — they were just never consulted at reclaim time (and shared memory didn't record which subscriber ids are reliable):

• ChannelControlBlock gains a reliable_subscribers bitset, maintained beside subscribers: registration records reliability before the subscriber becomes visible to publishers; server-side cleanup clears it so a dead subscriber can't gate publishers forever; placement-initialized at channel creation (like subscribers, since AtomicBitSet carries a runtime num_bits_).
• FindFreeSlotReliable stops at the first slot still pending in any reliable subscriber's available-slots bitset. Activation messages are exempt — subscribers skip them without claiming, so their delivery bits can remain set forever, and reclaiming them loses no data. kReadNewest skips remain deliberate skips (they clear the pending bits). The check only runs on the ring-full slow path, so nothing is added to the publish fast path.
• kChannelControlBlockVersion → 6 for the layout change.

Tests

• ReliableChannelIsLosslessForEveryReliableSubscriber — deterministic, single-threaded: fills the ring past a tight-reading reliable subscriber while a second reliable subscriber reads nothing; asserts the publisher backpressures (GetMessageBuffer returns null) instead of handing out the slow subscriber's backlog, that every message then arrives in order, and that the publisher resumes once both have caught up.
• ReliablePublisherUnblocksWhenLaggingSubscriberUnsubscribes — a lagging reliable subscriber gates the publisher; its departure releases the gate.

Both fail on unpatched main and pass with the fix; the full //client:client_test, //common:common_test, and //server:server_test suites pass with the fix (including ReliablePublisherDoesNotBlockOnUnreliableSubscriber and the other reliable-path tests).

We're carrying this as a patch against 3.0.3 in our tree in the meantime — happy to adjust whatever you'd like on naming, the version bump, or test placement.

[dallison]

Hi Matt, good to hear from you. Unfortunately I am in the UK for a while due to a family emergency. I can look at this in a couple of days if that's OK.

[dallison]

Matt, I think there is an already supported way to solve this. The subscriber has an option called keep_active_message that causes the subscriber to keep a reference to the most recently received message until you read another one. For a reliable subscriber this will keep a reliable reference to the most recently slot and will prevent it from being recycled. So a publisher won't reuse it and another subscriber will see it. Just set this on your subscribers and it should work as expected.

You can call ClearActiveMessage to remove the reference if you don't need it any more.

BTW, I got to this earlier than expected because I am jet lagged and can't sleep at the moment

[rudicus]

@dallison - Sorry to hear of your family troubles. Thanks for the tip on what already exists, I will give it a shot! Question though on what you mentioned just so I understand - (and no rush to respond) :

If two reliable subscribers (A and B) are both reading the same channel, and A reads the message first before B, it will set kMessageSeenByReliable on that message. If for whatever reason B is blocked (in our case a long running algorithm), A could keep reading new messages (and releasing the old ones with that bit set) before B reads it?

We are trying to do faster than realtime replay (64x) so certain things are super fast and could churn through the queue while we do have some algos that can't actually keep up that rate and would just like the system to "slow down" to match what we can actually handle

[rudicus]

Quick update: Your suggestion does seemingly resolve my issues! (Though not sure entirely how given the above scenario) Thanks. I can close this out if you like.

[dallison]

Glad it works.

If you have a really slow subscriber that is not keeping up with the messages and you are using reliable messages, then by definition the publisher will be unable to publish messages that will be missed by the slow subscriber. For reliable channels, the publisher will always be blocked if a subscriber will miss the message (like a nonblocking socket behavior in TCP).

So with reliable messages, you might get into a situation where the publisher is backpressured by the slowest subscriber. That is probably what you want.

[dallison]

I don't think we need this PR since it's already supported, but it might be worth adding some documentation since it's not obvious and the behavior of multiple publishers is subtle

Maybe it would be worth automatically setting the keep_active_message to true if reliable is set for subscribers to avoid the user having to set both options. That would avoid the subtle broken behavior. Reliable messaging isn't use much except for special cases like you are doing so it would be safe to set it automatically. A user can always clear it if they need to.

[mikael-s-persson]

Yeah, automatically setting it seems wise especially since the main purpose of having it off, afaik, is to economize slots, and nobody should really be trying to use reliable channels on a tight slot-budget anyway.

[dallison]

I thought about it overnight and it actually makes sense to always enable the keep_active_message feature if a subscriber is reliable since it's actually broken if you have more than one subscriber. So the feature would be enabled with reliable || keep_active_message.

[dallison]

Oh, I also thought that the best way to answer your question about why it works would be to ask Claude or GPT. They are very good at that.

[dallison]

Here's what Grok says:

A setting kMessageSeenByReliable does not mean the slot can be reused, and it is not “B has seen it too.” It only means some reliable subscriber has observed that message.

A can keep reading later messages. That only drops A’s refs on the old slots. The bit stays set; it is not a per-subscriber latch.

Reuse is decided oldest-first, and a reliable publisher will stop at the first slot that still has a reliable ref. A reliable subscriber that has already ReadMessage’d and is then stuck in a long algorithm still holds that current message (RetainsActiveMessage() is true for reliable subs). That slot stays pinned. The publisher cannot skip it, so it cannot overwrite that message or take later slots for new publishes.

What A can do is drain messages that are already in the channel. Those later slots can have kMessageSeenByReliable set and A’s refs gone, but they are not recycled while B’s older held slot is in the way. When B catches up, they are still there.

The risky case is B holding nothing on that message: never called ReadMessage, or ClearActiveMessage(). Then A seeing and releasing can leave refs == 0 with kMessageSeenByReliable already set, and the publisher is allowed to take the slot. Reclaiming it clears the available bit for every subscriber, including B.

So: A racing ahead does not steal a message B is still sitting on. The seen-by-reliable bit is only a first-visibility hint, not protection for the second subscriber.