p2p: Increase inbound capacity for block-relay only connections#28463
p2p: Increase inbound capacity for block-relay only connections#28463mzumsande wants to merge 7 commits into
Conversation
|
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers. Code Coverage & BenchmarksFor details see: https://corecheck.dev/bitcoin/bitcoin/pulls/28463. ReviewsSee the guideline for information on the review process.
If your review is incorrectly listed, please copy-paste ConflictsReviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first. |
Status QuoWe have a default maximum of 125 connections (not counting manual ones). Out of these, ProblemWe want to increase the number of inbound slots specifically offered to block-relay-only connections, but don't want to significantly change the number of tx-relaying inbound peers, because we don’t want to radically change the memory and traffic requirements that come with running a full node with the default configuration. ApproachWe propose to introduce a separate maximum of tx-relaying inbound connections, in addition to the existing global maximum. Changes to the eviction logicWe don't change the eviction logic for total connections, which happens in NumbersSpecifically, we propose to increase the total number of connections from 125 to 200, with a suggested ratio of 50% tx-relay inbounds. With 11 connection slots reserved for outbound peers, Handling bloom filter peersThe logic gets a bit complicated if a node supports BIP37 (bloom filters), because then an inbound peer can be switched to tx-relay after sending a |
|
Concept ACK, specifically on changes to the eviction logic, and handling bloom filters. I was curious whether there is (or will be) any other good reason to delay eviction after the version (apart from making this logic cleaner), but I haven't identified any. |
Discussed this with @amitiuttarwar and we think that it would be better to move the tx-related eviction into the version processing logic.
Cons:
I will update the PR with this soon! |
f78a0ed to
039ac99
Compare
039ac99 to
82ed8a5
Compare
|
Changed the approach to do the eviction of tx-relaying peers within the version message processing, when we can make use of the As a result of the later eviction, it became necessary to protect the new peer from being evicted right by |
82ed8a5 to
8247eaa
Compare
|
overall, I like this approach significantly more. although we are introducing another touchpoint for inbound eviction logic, it is more straightforward to reason about. but the biggest improvement is not unnecessarily disconnecting a new or existing connection when tx relay isn't being requested. reviewed this in-depth. the fundamental structure & logic looks good to me. left a lot of comments along the way 😛 some additional thoughts:
|
| * a peer to evict (all eligible peers are protected) | ||
| * so that the caller can deal with this. | ||
| */ | ||
| bool EvictTxPeerIfFull(std::optional<NodeId> protect_peer); |
There was a problem hiding this comment.
I wonder if it could be clearer to call this function something like TxInboundCapacityAvailable. imo that seems more useful from the POV of the net processing caller, but either is ok with me.
There was a problem hiding this comment.
default param here makes the FILTERLOAD/FILTERCLEAR callers slightly cleaner
| bool EvictTxPeerIfFull(std::optional<NodeId> protect_peer); | |
| bool EvictTxPeerIfFull(std::optional<NodeId> protect_peer) = std::nullopt; |
There was a problem hiding this comment.
changed to use default values.
Not sure about the rename suggestion, I think I prefer EvictTxPeerIfFull, because I think for naming describing what the function does is more important than describing the return value. Otherwise other spots might call this function in the future with the intention to just perform a check, and then might be surprised about the side effects.
There was a problem hiding this comment.
and then might be surprised about the side effects.
yeah totally. I was trying to make the return values more self-evident, but on further thought, I agree with you that emphasizing eviction is more important - people can always read the doc string to understand the return value. can't think of anything to concisely express HasOrMakeInboundTxCapacity, so leaving as is sounds good :)
8247eaa to
9555a33
Compare
|
Thanks for the review @amitiuttarwar! i pushed an update, addressing all feedback. |
df69b22 doc: improve documentation around connection limit maximums (Amiti Uttarwar) adc171e scripted-diff: Rename connection limit variables (Amiti Uttarwar) e9fd9c0 net: add m_max_inbound to connman (Amiti Uttarwar) c25e0e0 net, refactor: move calculations for connection type limits into connman (Amiti Uttarwar) Pull request description: This is joint work with amitiuttarwar. This has the first few commits of bitcoin#28463. It is not strictly a prerequisite for that, but has changes that in our opinion make sense on their own. It improves the handling of maximum numbers for different connection types (that are set during init and don’t change after) by: * moving all calculations into one place, `CConnMan::Init()`. Before, they were dispersed between `Init`, `CConnman::Init` and other parts of `CConnman`, resulting in some duplicated test code. * removing the possibility of having a negative maximum of inbound connections, which is hard to argue about * renaming of variables and doc improvements ACKs for top commit: amitiuttarwar: co-author review ACK df69b22 naumenkogs: ACK df69b22 achow101: ACK df69b22 Tree-SHA512: 913d56136bc1df739978de50db67302f88bac2a9d34748ae96763288d97093e998fc0f94f9b6eff12867712d7e86225af6128f4170bf2b5b8ab76f024870a22c
marcofleon
left a comment
There was a problem hiding this comment.
Concept ACK
Based on the discussion in #34542, there seems to be some general agreement that increasing block-relay-only connections is a good first step toward improving partition resistance. I think it makes sense on its own and it would also lay the groundwork for potentially adding reconciliation-based tx relay connections later on.
I’ll start running fuzzamoto and other relevant fuzz tests on this PR.
| } | ||
| pfrom.m_bloom_filter_loaded = true; | ||
| pfrom.m_relays_txs = true; | ||
| if (pfrom.IsInboundConn() && !m_connman.EvictTxPeerIfFull()) { |
There was a problem hiding this comment.
Is it intentional that we don't protect this new tx-relay peer like we do for the VERSION message in acfecb2? I guess this peer would have some history so it would make sense to consider it for eviction?
There was a problem hiding this comment.
I think it makes sense: the logic in the VERSION message follows the existing logic of rather replacing a peer than denying new peers if full (which helps against eclipse attacks).
I don't think that this is important for peers trying to get filters from us - those are likely to be only light clients anyway, and we don't gain much from that connection, so I don't think they need extra protection.
|
concept ACK. I would like some of the erlay discussion to move forward too, because I'm not sure how to judge this all holistically. |
|
Concept ACK. |
|
Concept ACK |
| } | ||
| } | ||
| if (tx_inbound_peers > m_max_inbound_full_relay) { | ||
| return AttemptToEvictConnection(/*evict_tx_relay_peer=*/true, protect_peer); |
There was a problem hiding this comment.
I ran mutation testing for this PR and noticed that the following mutant isn't killed by any test. See:
diff --git a/src/net.cpp b/src/net.cpp
index 499312fbbd..88afc05216 100644
--- a/src/net.cpp
+++ b/src/net.cpp
@@ -2504,7 +2504,7 @@ bool CConnman::EvictTxPeerIfFull(std::optional<NodeId> protect_peer)
}
}
if (tx_inbound_peers > m_max_inbound_full_relay) {
- return AttemptToEvictConnection(/*evict_tx_relay_peer=*/true, protect_peer);
+ return AttemptToEvictConnection(/*evict_tx_relay_peer=*/false, protect_peer);
}
:When evict_tx_relay_peer is true, it means that whether the node is over its tx-relay inbound limit, it must evict an existing tx-relay peer, not a block-relay-only peer. So, a test case to address it would be great. Suggestion (did not fully tested it):
diff --git a/test/functional/p2p_connection_limits.py b/test/functional/p2p_connection_limits.py
index 64dab1089e..56c1d81980 100755
--- a/test/functional/p2p_connection_limits.py
+++ b/test/functional/p2p_connection_limits.py
@@ -33,6 +33,22 @@ class P2PConnectionLimits(BitcoinTestFramework):
no_txrelay_version_msg.relay = 0
return no_txrelay_version_msg
+ def create_no_services_blocks_only_version(self):
+ """VERSION with relay=0 and nServices=0.
+ version_msg = msg_version()
+ version_msg.nVersion = P2P_VERSION
+ version_msg.strSubVer = P2P_SUBVERSION
+ version_msg.nServices = 0
+ version_msg.relay = 0
+ return version_msg
+
def test_inbound_limits(self):
node = self.nodes[0]
@@ -81,6 +97,18 @@ class P2PConnectionLimits(BitcoinTestFramework):
node.add_p2p_connection(P2PInterface())
self.wait_until(lambda: len(node.getpeerinfo()) == 2)
+ self.log.info('Test that EvictTxPeerIfFull only evicts tx-relaying peers')
+ NUM_BLOCK_RELAY_PEERS = 21
+ self.restart_node(0, ['-maxconnections=33', '-inboundrelaypercent=0'])
+ for _ in range(NUM_BLOCK_RELAY_PEERS):
+ p = self.nodes[0].add_p2p_connection(P2PInterface(), send_version=False, wait_for_verack=False)
+ p.send_without_ping(self.create_no_services_blocks_only_version())
+ self.wait_until(lambda: len(node.getpeerinfo()) == NUM_BLOCK_RELAY_PEERS)
+
+ with node.assert_debug_log(['failed to find a tx-relaying eviction candidate - connection dropped'], timeout=5):
+ self.nodes[0].add_p2p_connection(P2PInterface(), expect_success=False, wait_for_verack=False)
+ self.wait_until(lambda: len(node.getpeerinfo()) == NUM_BLOCK_RELAY_PEERS)
+
if __name__ == '__main__':
P2PConnectionLimits(__file__).main()There was a problem hiding this comment.
Thanks, good find. I added a commit with your suggested test, just adding a short comment why nServices=0.
|
🚧 At least one of the CI tasks failed. HintsTry to run the tests locally, according to the documentation. However, a CI failure may still
Leave a comment here, if you need help tracking down a confusing failure. |
|
I have a node with a lot of inbound connections if you want help testing... ipv4 ipv6 onion total block obviously each time I restart it I lose them all but normally bounces back after a few days. |
I've been running two nodes with this PR (mzumsande@cfa72f8) for a while. Running this Node [
{
"key": "block-relay-only / relaytxes=false",
"count": 2
},
{
"key": "inbound / relaytxes=false",
"count": 45
},
{
"key": "inbound / relaytxes=true",
"count": 94
},
{
"key": "outbound-full-relay / relaytxes=true",
"count": 8
}
]Node [
{
"key": "block-relay-only / relaytxes=false",
"count": 2
},
{
"key": "inbound / relaytxes=false",
"count": 42
},
{
"key": "inbound / relaytxes=true",
"count": 94
},
{
"key": "outbound-full-relay / relaytxes=true",
"count": 8
}
]Compared to a node [
{
"key": "block-relay-only / relaytxes=false",
"count": 2
},
{
"key": "inbound / relaytxes=false",
"count": 26
},
{
"key": "inbound / relaytxes=true",
"count": 87
},
{
"key": "outbound-full-relay / relaytxes=true",
"count": 8
}
] |
|
will review actively once feedback is addressed, as promised :) |
|
Concept ACK |
|
0e5c40e to 6b2fda0: added test by @brunoerg Thanks for the reviews and sorry for the delay. I think I have addressed all feedback now!
Thanks! Since @0xB10C has been running this for some time now and (to my knowledge) no issues have come up, there is nothing specific that needs testing - though just runing a node with this branch and reporting anything strange will help! |
My thinking is that Erlay and this PR are two different ways of increasing connectivity, and we should probably only do one at a time, even if they are technically not exclusive. With Erlay we would also need to increase Erlay connections have the additional benefit of also helping against tx censorship, whereas this PR only strengthens against partitioning/eclipse attacks, so Erlay does more. The downside is the added complexity of Erlay and the added traffic / reconciliation efforts. |
|
@mzumsande we talked offline and the next step for erlay is seeing what reconciliation-only connections would look like. We can move on with this effort for now. |
instagibbs
left a comment
There was a problem hiding this comment.
initial comments as I dig in a bit
| * @param[in] protect_peer Protect peer with node id | ||
| * @return True if a node was marked for disconnect | ||
| */ | ||
| bool AttemptToEvictConnection(bool evict_tx_relay_peer = false, std::optional<NodeId> protect_peer = std::nullopt) EXCLUSIVE_LOCKS_REQUIRED(!m_nodes_mutex); |
There was a problem hiding this comment.
asking only: might be worth at least making the first arg non-optional?
There was a problem hiding this comment.
I did that. I also renamed it to evict_tx_relay_peer_only so that it becomes clear that if set to false, a tx peer may still be evicted.
| } | ||
|
|
||
| // If we have too many tx-relaying inbound peers, attempt to evict an existing one. | ||
| // Only if this fails, disconnect this peer. |
There was a problem hiding this comment.
Given this new callsite which may also result in a AttemptToEvictConnection, could one connection cause two disconnects? Perhaps disconnect a blocksonly on the first one, the a tx relay on second? Trying to understand what's possible.
There was a problem hiding this comment.
Yes, I think that this would be possible. I'm not sure if it's much of a problem though.
Since we don't know whether the peer wants tx relay in CreateNodeFromAcceptedSocket, I'm not sure how we could avoid it.
| @@ -561,7 +561,11 @@ void SetupServerArgs(ArgsManager& argsman, bool can_listen_ipc) | |||
| argsman.AddArg("-forcednsseed", strprintf("Always query for peer addresses via DNS lookup (default: %u)", DEFAULT_FORCEDNSSEED), ArgsManager::ALLOW_ANY, OptionsCategory::CONNECTION); | |||
| argsman.AddArg("-listen", strprintf("Accept connections from outside (default: %u if no -proxy, -connect or -maxconnections=0)", DEFAULT_LISTEN), ArgsManager::ALLOW_ANY, OptionsCategory::CONNECTION); | |||
| argsman.AddArg("-listenonion", strprintf("Automatically create Tor onion service (default: %d)", DEFAULT_LISTEN_ONION), ArgsManager::ALLOW_ANY, OptionsCategory::CONNECTION); | |||
There was a problem hiding this comment.
in commit message: "As a result, the tx-related maximum traffic should not change
drastically." is there a particular reason we can't shoot for parity?
There was a problem hiding this comment.
With "parity", do you mean choose parameters such that the maximum tx-related traffic is exactly the same as now?
We could do that, but I think having "nice" numbers such as 200 max connections and FULL_RELAY_INBOUND_PCT=50 is more useful.
I am already a bit worried that changing the long-existing number of 125 maximum connections to a mix of two maxima might confuse users. It would be nice if at least the new values would be something humans can remember.
| m_max_outbound_block_relay = std::min(MAX_BLOCK_RELAY_ONLY_CONNECTIONS, m_max_automatic_connections - m_max_outbound_full_relay); | ||
| m_max_automatic_outbound = m_max_outbound_full_relay + m_max_outbound_block_relay + m_max_feeler; | ||
| m_max_inbound = std::max(0, m_max_automatic_connections - m_max_automatic_outbound); | ||
| m_max_inbound_full_relay = std::max(0, static_cast<int>(FULL_RELAY_INBOUND_PCT / 100.0 * m_max_inbound)); |
There was a problem hiding this comment.
IIUC at low "maxconnections" values, the int math flooring could result in having no inbound when the argument would otherwise suggest it could have some. Not saying this is worth "fixing" since people generally don't enable listening and reduce the number drastically, just noting
If the added bool evict_tx_relay_peer_only is set, non-tx-relay peers will be exempt from disconnection. Also allow to specify a peer that exempt from eviction. Both options will be used in later commits. Co-authored-by: Amiti Uttarwar <amiti@uttarwar.org>
..and adjust the eviction logic. The new default max connection number is 200, the default maximum of tx-relaying inbounds is limited to 50% of all inbound connections. With 11 outbound connections, that is (200 - 11) * 0.5 = 94.5. As a result, the tx-related maximum traffic should not change drastically. When we receive an inbound connection and don't have space for another full-relay peer, we now attempt to evict specifically a full-relay inbound after receiving the version message of the new peer. Once this commit is widely deployed, the added inbound capacity will allow us to increase the number of outgoing block-relay-only connections. Co-authored-by: Amiti Uttarwar <amiti@uttarwar.org>
Co-authored-by: Martin Zumsande <mzumsande@gmail.com>
Permit users to change the amount of inbounds that are permitted to relay transactions. This is particularly relevant to ensure that superusers that are not concerned with resource usage are not artificially restricted from offering many transaction relay slots to the network. Co-authored-by: Martin Zumsande <mzumsande@gmail.com>
… a peer to tx relay Co-authored-by: Amiti Uttarwar <amiti@uttarwar.org>
Co-authored-by: Martin Zumsande <mzumsande@gmail.com>
|
I've been running dbdddbc (from June 4th) for a few weeks and switching the nodes to something else for now (#35465 (comment)). Some numbers, similar to #28463 (comment): On On node |
| { | ||
| ServiceFlags m_local_services = NODE_NONE; | ||
| int m_max_automatic_connections = 0; | ||
| int m_full_relay_inbound_percent = 0; |
There was a problem hiding this comment.
I think for fuzz tests like p2p_handshake and process_message that initialize connman with the testing setup defaults, initializing to 0 for both max connections and the inbound percentage could be an issue?
The tests add peers to connman.m_nodes before doing a version handshake, which now calls EvictTxPeerIfFull(). Wouldn't it just disconnect a test inbound peer with m_relay_txs=true right away?
If they do need to be set to 0 here, then maybe we should override them in TestingSetup in setup_common.cpp.
| { | ||
| LOCK(m_nodes_mutex); | ||
| for (const CNode* pnode : m_nodes) { | ||
| if (pnode->IsInboundConn() && pnode->m_relays_txs) { |
There was a problem hiding this comment.
Match the other functions like GetExtraFullOutboundCount/GetExtraBlockRelayCount?
| if (pnode->IsInboundConn() && pnode->m_relays_txs) { | |
| if (pnode->IsInboundConn() && !pnode->fDisconnec &&pnode->m_relays_txs) { |
| { | ||
| ServiceFlags m_local_services = NODE_NONE; | ||
| int m_max_automatic_connections = 0; | ||
| int m_full_relay_inbound_percent = 0; |
| argsman.AddArg("-maxconnections=<n>", strprintf("Maintain at most <n> automatic connections to peers (default: %u). This limit does not apply to connections manually added via -addnode or the addnode RPC, which have a separate limit of %u. It does not apply to short-lived private broadcast connections either, which have a separate limit of %u.", DEFAULT_MAX_PEER_CONNECTIONS, MAX_ADDNODE_CONNECTIONS, MAX_PRIVATE_BROADCAST_CONNECTIONS), ArgsManager::ALLOW_ANY, OptionsCategory::CONNECTION); | ||
| argsman.AddArg("-maxconnections=<n>", strprintf("Maintain at most <n> automatic connections to peers (default: %u). %u slots of these are reserved for outgoing connections. See -inboundrelaypercent for more information about limits applied to transaction relay inbound peers. " | ||
| "This limit does not apply to connections manually added via -addnode or the addnode RPC, which have a separate limit of %u." | ||
| "It does not apply to short-lived private broadcast connections either, which have a separate limit of %u.", |
There was a problem hiding this comment.
| "It does not apply to short-lived private broadcast connections either, which have a separate limit of %u.", | |
| "It does not apply to short-lived private broadcast connections either, which have a separate limit of %u. ", |
|
|
||
| // If we have too many tx-relaying inbound peers, attempt to evict an existing one. | ||
| // Only if this fails, disconnect this peer. | ||
| if (pfrom.IsInboundConn() && pfrom.m_relays_txs) { |
There was a problem hiding this comment.
might be confused but: If we're running in blocksonly mode, should we be dropping inbounds because they say it's ok to announce txs to them (but we won't)
| pfrom.m_bloom_filter_loaded = false; | ||
| pfrom.m_relays_txs = true; | ||
| if (pfrom.IsInboundConn() && !m_connman.EvictTxPeerIfFull()) { | ||
| // We don't have room for another tx-relay peer, disconnect |
There was a problem hiding this comment.
nit: could these 3 locations where we're filtering get a helper function to increase grep-ability? ala MaybeDisconnectForTxRelayCapacity?
This is joint work with amitiuttarwar.
See issue #28462 for a broader discussion on increasing the number of block-relay-only connections independent of this particular implementation proposal.
We suggest to increase the number of inbound slots allocated to block-relay-only peers by increasing the default maximum connections from 125 to 200, with 50% of inbound slots accessible for tx-relaying peers.
This is a prerequisite for being able to increase the default number of outgoing block-relay-only peers later, because the current inbound capacity of the network is not sufficient.
In order to account for incoming tx-relaying peers separately from incoming block-relay peers, changes to the inbound eviction logic are necessary.
See the next post in this thread for a more detailed explanation and motivation of the changes.