Repository navigation
[bug]: htlc_maximum_msat is not optional anymore in channel updates #7290
Description
Background
lnd forwards channel updates without htlc_maximum_msat which is now forbidden after lightning/bolts#999 was merged.
The problem is that in this case, other nodes (cln for instance) send back a warning message which triggers a disconnect.
Your environment
lnd v0.15.5-beta or master
Steps to reproduce
Run lnd and create a channel to a node running cln.
Expected behaviour
No channel update messages without htlc_maximum_msat
Actual behaviour
Some channel update messages without htlc_maximum_msat. Then a warning is issued by the other node (cln for instance). Then the peer is disconnected in lnd v0.15.5 because the warning message is "unknown" and in master because a warning triggers a disconnect for an active channel.
Activity
yaslama commentedon Jan 5, 2023
BTW the logs were the same as #7273 and we backported the warning message handling from master to see that the problem was the issue described here
kingonly commentedon Jan 5, 2023
@saubyk this is spec incompatibly issue that has some nasty side effects. Can you please consider prioritizing to the next update?
Roasbeef commentedon Jan 5, 2023
Interesting, I think another operative question is: what software is sending out update w/o max sat set? FWIW, the spec doesn't say precisely what to do if you encounter such an update. A conservative thing to do would just be to forward it anyway, since it's older software that may not be keeping up lockstep with the frequent spec changes. If we started to reject this wholesale, this could lead to other unintended propagation issues (we've fixed a number of them in the upcoming 0.16 release).
This should have been resolved with this PR: #6840 -- are you able to repro?
kingonly commentedon Jan 5, 2023
We don't know what software is sending these updates, but we're seeing many.
I think messages that don't comply with the spec shouldn't be propagated. Are there any more that are being propagated?
Roasbeef commentedon Jan 5, 2023
I think this is the CLN commit that starts to reject (parsed as zero so fails validation?): ElementsProject/lightning@5553b85
I looked at this Eclair PR, but can't tell for sure if they're rejecting these updates themselves: https://github.com/ACINQ/eclair/pull/2299/commits
Looks like RL has been doing this since July or so (before that spec PR above was merged): https://github.com/lightningdevkit/rust-lightning/pull/1519/files
Gotcha, that's interesting, since the original motive for this was that from observations pretty much no updates were setting the value. If we're still seeing a lot, then I think that argues for pushing out the deployment of the spec modification.
From the original spec PR:
Generally I'd agree, but this is a situation where something in the spec was merged that affects gossip, but not everyone has started to observe the new rules. If there're a lot of updates from routing nodes that would no longer propagate with strict enforcement, then we might want to consider holding off if the old assumptions weren't correct (tho ofc the change is trivial for the most part).
I guess w/e unknown impl/version is sending update w/o it set will just be slowly cut off from the network 🤷
yaslama commentedon Jan 6, 2023
The peer was disconnected until I changed the line https://github.com/lightningnetwork/lnd/blob/master/peer/brontide.go#L1705 to return false instead of true (or commented the case).
yaslama commentedon Jan 6, 2023
In our logs, the channel updates without htlc_maximum_msat are very old (from 2019). In fact anyone can run a fake node which send such channel updates. I think that it's better to ignore any such invalid message and not forward them if we are sure that the last stable version(s) of the main known implementations are not generating them.
Crypt-iQ commentedon Jan 26, 2023
Can you re-link but with a "permalink" - that link now points to a comment since
masterchanged. I can't see how eithermasterorv0.15.5-betareceiving awarningresults in disconnect. I think the CLN node here is sending awarningand then immediately disconnecting.Do you have logs here? Or does the
warningmessage give any more info? That would be very helpfulCrypt-iQ commentedon Jan 31, 2023
We confirmed after testing with a live CLN node that this is because the
channel_updatesent to CLN didn't have themust_be_onebit set in themessage_flagsbitfield: https://github.com/lightning/bolts/blob/master/07-routing-gossip.md#the-channel_update-messageTo fix this issue, we need to make sure that they 1) no longer send out
ChannelUpdatewith the bit un-set and 2) no longer acceptChannelUpdatewith the bit un-set.Point 1) is necessary because current nodes may already have these
ChannelUpdatesin their graph. If they were to perform a historical sync with a CLN node, they may send one of these olderChannelUpdatesto the CLN node. There are six places that we send outChannelUpdates:ChannelUpdatehere)retransmitStaleAnnscase should catch it)d.reliableSender.sendMessagein the gossiper:PropagateChanPolicyUpdatecall-path, we already perform an on-the-fly migration to set the bit here:lnd/routing/localchans/manager.go
Lines 215 to 216 in 5d22d5e
ChannelUpdatestored in thereliableSenderPoint 2) is so that we align with the spec and no longer accept the legacy encoding. There are two places where we process
ChannelUpdate:It may be possible to handle point 1) by simply performing a graph migration that checks for all of our own legacy
ChannelUpdateand then adding checks to the reliable sender to ensure that we don't accidentally send out a legacyChannelUpdatedespite this migration.As discussed offline, for point 2), we don't need to disconnect a peer for sending us this legacy
ChannelUpdate. Instead, we can just drop theChannelUpdateto ensure we don't segment older LND nodes. In the future, we can revisit this logic and potentially disconnect.