validation: improve block data I/O error handling in P2P paths#35003
validation: improve block data I/O error handling in P2P paths#35003furszy wants to merge 5 commits into
Conversation
|
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers. Code Coverage & BenchmarksFor details see: https://corecheck.dev/bitcoin/bitcoin/pulls/35003. ReviewsSee the guideline and AI policy for information on the review process.
If your review is incorrectly listed, please copy-paste ConflictsReviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first. LLM Linter (✨ experimental)Possible typos and grammar issues:
Possible places where named args for integral literals may be used (e.g.
Possible places where comparison-specific test macros should replace generic comparisons:
2026-07-02 15:13:56 |
23b57a9 to
5a0191e
Compare
5a0191e to
8c1b0d6
Compare
8c1b0d6 to
72f7a67
Compare
|
concept ACK |
frankomosh
left a comment
There was a problem hiding this comment.
tACK 72f7a67
Built from source. All new tests pass:
flatfile_tests: 9 cases, no errorsvalidation_chainstate_tests: 5 cases, no errorsblockmanager_tests: 8 cases, no errorsp2p_handle_io_errors.py: both scenarios pass
Also manually reverted the GETBLOCKTXN fix (commit ba034f4) by removing the fatalError call and replacing with a silent return. The functional test correctly caught this kind of mutation. the test expected the node to shut down, but the mutant node stayed alive, causing a timeout failure. Mutant killed by p2p_handle_io_errors.py. Unit tests did not detect this mutation. Also process_messages and process_message fuzz harnesses both passed without detecting the mutation, confirming the functional test is providing the critical oracle here.
Line 5915 in SendMessages has assert(ret) after a ReadBlock call in the compact block announcement path. I think this is the same pattern as the GETBLOCKTXN assert fixed in commit 4. I believe this is the "another assertion scenario in compact block relay" mentioned in the PR description. Is this planned as a follow-up? Happy to take it if so.
maflcko
left a comment
There was a problem hiding this comment.
concept ack, left a test suggestion, which could make review easier
72f7a67 to
f0ce04b
Compare
f0ce04b to
5a25f94
Compare
Yes @frankomosh. All yours :). |
5a25f94 to
fc86ae4
Compare
| block_hash = node.getblockhash(3) # Not in recent cache | ||
| with simulate_io_error(node.blocks_path): | ||
| # Bug: we currently swallow the GETDATA request, never reply to the other party, and log issue under debug-NET | ||
| with node.assert_debug_log(expected_msgs=["[ProcessMessages] [net] ProcessMessages(getdata, 37 bytes): Exception 'filesystem error"], |
There was a problem hiding this comment.
Note: intentionally not checking for a specific filesystem error msg, since it slightly vary across OS. Specific msg doesn't matter, this first commit just proves the error is swallowed in master.
fc86ae4 to
825dfc2
Compare
maflcko
left a comment
There was a problem hiding this comment.
Looks like the test fails on the second commit, or so?
Also, provided some more test nits. Feel free to ignore.
825dfc2~4 📭
Show signature
Signature:
untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
trusted comment: 825dfc200520d503fec64346ed01a8b12ec8fa54~4 📭
HuyWCmP9dlHdaEd4PL2XkITSC+f4T6nxsowBmRtzkzVkthldhUIEhYd8dWE2Z+BGoWVYLPWjRRxIPvoLUhqCCg==
| self.setup_clean_chain = True | ||
|
|
||
| def test_block_connect_on_broken_fs(self): | ||
| self.log.info("Test block connect on inaccessible filesystem") |
There was a problem hiding this comment.
nit in 75a8a95:
From the pull description:
If it happens during block connection, after AcceptBlock(), inside ActivateBestChain(), the file system error gets thrown and swallowed by the P2P general try-catch, leaving the node in a living but stuck state. Where ActivateBestChain() will always silently fail on any posterior block processing, retrying to active the failing block.
You claim that a failure in ActivateBestChain will leave the node alive, but stuck. However, when writing a test for this, I found that it properly shut down:
diff --git a/test/functional/p2p_handle_io_errors.py b/test/functional/p2p_handle_io_errors.py
index 4438ad1dd5..4706dfebd4 100755
--- a/test/functional/p2p_handle_io_errors.py
+++ b/test/functional/p2p_handle_io_errors.py
@@ -14,2 +14,3 @@ from test_framework.blocktools import (
create_block,
+ create_empty_fork,
)
@@ -49,5 +50,6 @@ class P2PBlockIOErrorTest(BitcoinTestFramework):
self.setup_clean_chain = True
+ self.extra_args = [['-fastprune']]
- def test_block_connect_on_broken_fs(self):
- self.log.info("Test block connect on inaccessible filesystem")
+ def test_block_accept_on_broken_fs(self):
+ self.log.info("Test block accept on inaccessible filesystem")
node = self.nodes[0]
@@ -70,2 +72,24 @@ class P2PBlockIOErrorTest(BitcoinTestFramework):
+ def test_block_connect_on_broken_fs(self):
+ self.log.info("Test block connect on inaccessible filesystem")
+ node = self.nodes[0]
+ peer = node.add_p2p_connection(P2PInterface())
+ blocks = create_empty_fork(node, fork_length=2)
+ large_block = self.generatetodescriptor(node, 1, f"raw({'55'*100_000})")[0]
+ peer.send_and_ping(msg_block(blocks[0]))
+ (node.blocks_path / 'blk00002.dat').unlink()
+ node.getblock(large_block)
+
+ with node.assert_debug_log(expected_msgs=["ActivateBestChain failed (Failed to read block.)"]):
+ peer.send_without_ping(msg_block(blocks[1]))
+ peer.wait_for_disconnect()
+
+ node.wait_until_stopped(
+ expect_error=True,
+ expected_stderr=re.compile(r"fatal internal error"),
+ )
+
+ # Restart node for next test
+ self.start_node(0, extra_args=['-reindex'])
+
def test_getdata_on_broken_fs(self):
@@ -125,2 +149,3 @@ class P2PBlockIOErrorTest(BitcoinTestFramework):
+ self.test_block_accept_on_broken_fs()
self.test_block_connect_on_broken_fs()I guess it could make sense to add the test?
Also, it could make sense to clarify that point 2 in the pull request description refers to a thrown exception, presumably only from create_directories, and does not refer to any IO error?
Just a nit, and up to you, if you want to keep this pull focussed only on the create_directories call, or be a more general io error handling improvement in P2P paths, with test coverage.
There was a problem hiding this comment.
You claim that a failure in ActivateBestChain will leave the node alive, but stuck. However, when writing a test for this, I found that it properly shut down.
Yeah ok. Your test exercises a missing block file, not the inability to access the directory. If we want to replicate the stuck, the goal there should be to get the node stuck by making it throw inside the create_directories call.
I had a test early in this branch that reproduced the "throw inside ActivateBestChain" through the P2P path in a deterministic way (which is not easy due to AcceptBlock happening first). But I did not like that it relied on another bug in reconsiderblock to trigger this one.
Basically, the approach was to invalidate the tip, submit a fork, make the directory inaccessible, and then reconsider the block, which throws internally but resets the block validity flags (the operation is not atomically reverted upon failure). After that, sending any number of descendant blocks via P2P will be swallowed due to the inability to activate the very first block. But it was ugly to depend on reconsiderblock to trigger it.
I guess it could make sense to add the test?
Yes! for sure. Pulling it.
Also, it could make sense to clarify that point 2 in the pull request description refers to a thrown exception, presumably only from create_directories, and does not refer to any IO error?
Sure. Will do.
There was a problem hiding this comment.
I had a test early in this branch that reproduced the "throw inside
ActivateBestChain" through the P2P path in a deterministic way (which is not easy due toAcceptBlockhappening first). But I did not like that it relied on another bug inreconsiderblockto trigger this one.Basically, the approach was to invalidate the tip, submit a fork, make the directory inaccessible, and then reconsider the block, which throws internally but resets the block validity flags (the operation is not atomically reverted upon failure). After that, sending any number of descendant blocks via P2P will be swallowed due to the inability to activate the very first block. But it was ugly to depend on
reconsiderblockto trigger it.
Ah interesting. I presume the bug still exists after this pull requests, as the ReconsiderBlock function is not changed here? Just wondering, as the bug seems unrelated to the bug fixed in this pull request. Also, I agree that such a test may be too spicy to include here.
There was a problem hiding this comment.
I presume the bug still exists after this pull requests, as the ReconsiderBlock function is not changed here?
Would say it is partially fixed but haven't tried it. ReconsiderBlock internally calls ActivateBestChain, which triggers a fatal error after this PR, instead of throwing an exception. So it is a matter of whether we are flushing the block flags reset to disk prior to aborting or not. Because, if we do, the node would fail early on any following up startup.
It seems the assertion error message is libc-dependent. Will make the |
825dfc2 to
476fc0f
Compare
|
Haven't read all of the discussion threads, but since my last review, the only change seems to be moving the chmod, which doesn't seem like a good change. (See inline comment) Also, the suggestion by @josibake to use exceptions here sounds interesting. However, I haven't looked at the details and code, so it is harder to tell which is the right fit here. I'd be happy to review either approach, and wonder what other reviewers prefer here? lgtm ACK d4f9539 🐻 Show signatureSignature: |
| old_mode = stat.S_IMODE(os.stat(parent_dir).st_mode) | ||
|
|
||
| os.rename(blocks_path, blocks_bak) | ||
| os.chmod(parent_dir, 0o500) # Prevent re-creation of blocks/ |
There was a problem hiding this comment.
#35003 (comment): I don't think this change was a beneficial change. When the chmod fails, the test should also fail, not silently pass and even skip running the fn/yield.
I think the correct fix would be to revert this change and then wait for the test to fail, and then figure out if that env/platform should be skipped or supported.
d4f9539 to
81cddf0
Compare
|
I hopefully tackled all remaining feedback. Also added coverage for the |
l0rinc
left a comment
There was a problem hiding this comment.
code review ACK 81cddf0
The changes look correct to me and the new coverage exercises the relevant failure paths well.
Seems to me the remaining compact-block announcement should likely get the same fatal-error treatment, but I'm also fine with handling it as a follow-up.
(nit: PR description needs slight adjustment now regarding commit order)
| // pruned after we release cs_main above, so this read should never fail. | ||
| assert(ret); | ||
| // Unless an I/O error occurs, in which case we want to shut down gracefully. | ||
| if (!ret) { |
There was a problem hiding this comment.
It seems to me compact-block announcements still abort on the same kind of failure:
bitcoin/src/net_processing.cpp
Lines 5997 to 5998 in dc282ff
Is that deliberate, or could we use the same fatal-error style there?
nit: please see my suggestion above on how we can simplify the code after the assertion removal.
Yes, it is in the PR description. You probably forgot this one #35003 (review) and my reply #35003 (comment). |
maflcko
left a comment
There was a problem hiding this comment.
only test changes since last review, re-ACK 81cddf0 🌊
Show signature
Signature:
untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
trusted comment: only test changes since last review, re-ACK 81cddf0d878a27b84c85db831be0165648b89e94 🌊
25kQW99wcgHBdEbUId8fCqwmWawZrgdJHnNKS3hwKbmceslmlwVE98zG84KPEXu7zjtO4F8GL1Yeaj9ewBhNAQ==
|
|
||
| @contextlib.contextmanager | ||
| def simulate_io_error(target_path): | ||
| # Make files under target_path inaccessible by renaming its parent dir, |
There was a problem hiding this comment.
in cfdcd3b: I don't think this recent force push is correct.
Not the target_path's parent is renamed, but the target_path itself.
| node.getblockcount() | ||
|
|
||
| def test_getcfilters_on_broken_fs(self): | ||
| """GETCFILTERS must log the filter-index read failure and keep running.""" |
There was a problem hiding this comment.
in the first commit: Just like getdata, it would be kind to the peer to disconnect it, rather than silently not reply to the request?
There was a problem hiding this comment.
I was thinking about tackling it in a separate PR because it would be good to cover all filter-related messages at once, not just this one. We disconnect inside PrepareBlockFilterRequest for all of them, but not in any of the subsequent lookups.
Add tests showing that block disk I/O failures triggered
through P2P requests are silently swallowed.
The idea is to simulate I/O errors by making the blocks directory
temporarily inaccessible while the node is running. This allows block
reads to fail when serving P2P requests.
Aside from the well behaving test cases, this covers a few cases
where the node mishandles the error:
- GETDATA: failures should be handled locally by disconnecting the
peer and logging the error, without impacting node operation.
- GETBLOCKTXN: failures when accessing recent blocks violate our
"last 10 blocks must always be available for reorgs" rule and
must result in a fatal error, rather than being caught and only
logged by the P2P message handling general try-catch.
- txindex/txospenderindex: getrawtransaction and gettxspendingprevout
open the block file directly, so filesysteme exceptions leak out as
a generic RPC error instead of being handled cleanly.
- GETCFILTERS: failures should be logged through the filter index read
path and leave the node running, rather than being swallowed by the
P2P messages general try-catch.
Co-authored-by: MarcoFalke <*~=`'#}+{/-|&$^_@721217.xyz>
The callers of this function don't expect it to throw. For example, when the blocks directory becomes inaccessible (e.g. volume detach or permission changed), FlatFileSeq::Open's call to fs::create_directories() throws filesystem_error. On the block connection path, if this occurs during ActivateBestChain(), just after AcceptBlock() the exception bypasses FatalError entirely. It gets caught by ProcessMessage's general catch-all with only a NET-level debug log, swallowing the error msg for any regular user, and every subsequent block arrival retries the same failing read, leaving the node looking healthy while the block processing mechanism is stuck.
Cover the two ActivateBestChain paths that read data from disk: ConnectTip (block connection) and DisconnectTip (reorg). Simulate inaccessible block and undo files, ensure fatal error and shutdown is triggered.
Verifies that BlockManager behaves the same way whether a block file is missing or the blocks directory is inaccessible. Reads (ReadBlock, ReadRawBlock, ReadBlockUndo) return false, writes (WriteBlock, WriteBlockUndo) return null/false, and failures are logged. The goal is consistent failure handling. Previously, missing block files returned false and logged an error, while an inaccessible blocks directory was throwing an exception.
…tdown A failure here indicates a serious inconsistency or disk issue, and the current behavior is to abort immediately, which is correct but not the best. Replace assert with a fatal error so the node shuts down through the normal shutdown path instead. This does not change the outcome (the node still stops), but allows a more controlled teardown, giving other components a chance to flush their state even when a particular block (or the blocks dir) on disk is unreachable. E.g. the blocks dir might be the only affected, the mempool might be ok, and we don't want to lose such information.
81cddf0 to
7680140
Compare
|
reACK 7680140 |
|
test-doc-only re-ACK 7680140 🍏 Show signatureSignature: |
|
@rkrux @w0xlt @frankomosh @josibake can you take another look here? |
|
Approach NACK While I certainly appreciate the work @furszy has put into this, the PR in its current form is not the direction I think we should be going. I spent some time reading and thinking through a more comprehensive error handling strategy for the codebase in order to convince myself I'm not pushing back simply for aesthetics, and I am convinced removing exceptions from the low level file primitive and folding it into a I would much prefer something like this in, instead of the index b060108c18..df89984d53 100644
--- a/src/node/blockstorage.cpp
+++ b/src/node/blockstorage.cpp
@@ -821,13 +821,23 @@ void BlockManager::UnlinkPrunedFiles(const std::set<int>& setFilesToPrune) const
AutoFile BlockManager::OpenBlockFile(const FlatFilePos& pos, bool fReadOnly) const
{
- return AutoFile{m_block_file_seq.Open(pos, fReadOnly), m_obfuscation};
+ try {
+ return AutoFile{m_block_file_seq.Open(pos, fReadOnly), m_obfuscation};
+ } catch (const fs::filesystem_error& e) {
+ LogError("OpenBlockFile failed for %s: %s", pos.ToString(), e.what());
+ return AutoFile{nullptr, m_obfuscation};
+ }
}
/** Open an undo file (rev?????.dat) */
AutoFile BlockManager::OpenUndoFile(const FlatFilePos& pos, bool fReadOnly) const
{
- return AutoFile{m_undo_file_seq.Open(pos, fReadOnly), m_obfuscation};
+ try {
+ return AutoFile{m_undo_file_seq.Open(pos, fReadOnly), m_obfuscation};
+ } catch (const fs::filesystem_error& e) {
+ LogError("OpenUndoFile failed for %s: %s", pos.ToString(), e.what());
+ return AutoFile{nullptr, m_obfuscation};
+ }
}
fs::path BlockManager::GetBlockPosFilename(const FlatFilePos& pos) constThis preserves the exception, and moves the translation of that error into What my proposal does not address is the I do understand that the approach in this PR was done with consistency in mind, but I don't think that is enough of a justification when the thing being made consistent is, as I've tried to argue here, a bad design. |
| fs::create_directories(path.parent_path()); | ||
| const fs::path path = FileName(pos); | ||
| // Create parent path only if we are going to write. | ||
| if (!read_only) { |
There was a problem hiding this comment.
Note: this is unrelated to error handling afaict and could be pulled out into its own PR with its own test, no?
|
Let's recall this pull has more than 150 comments and is sitting for 3 months (which is reasonable for its size). However, it doesn't seem to be getting traction and has outstanding issues, some of which are enough to NACK it. So, I don't think the pull will be merged as-is. Even if it was merged, there'd be a bunch of follow-ups, so it would be good to decide how to address them before merge. I think it would be good to step back and answer a meta question:
There are different answers:
|
|
Thanks for zooming out @maflcko , I think its helpful. I'll respond here, but this is starting to feel like a discussion that can be moved to an issue around error handling in general, considering the discussion now spans two PRs.
This is a policy question. I think its up to the caller to decide, which is why I've been pushing back on folding multiple errors into a nullptr, and pushing back on having the low level function abort instead of propagating the error up. Said differently, the answer to this question will change in the future, and it will likely be different for different callers.
I think we should stop swallowing errors with broad catch alls that do nothing. Instead, if we can't catch it, let it terminate. If we would prefer a graceful fatal shutdown instead, have broad catch all that executes a safe shut down. Even better, have each boundary own their own error handling policy as I recommend here: BlockManager should own storage errors, BlockfilterIndex should manage storage errors related to indexes, etc etc. In each of these boundaries, if they have the information they need to know that a graceful fatal shutdown is possible/safe/etc, they can do that. If the cant make that call, they should allow the error to propagate. |
|
Restating the discussion we already had: #35003 (comment). Today, every filesystem error in Overall, following the by the books approach without pondering the actual sources is just bad in my view. And treating this bug fix PR as a broader architectural design discussion spot, instead of just creating a new issue or PR (as it was suggested in my first replies to you), is just blocking progress unnecessarily.
The PR is 97% about adding tests. It changes 5 lines of code in the node at most. |
Sorry for being unclear, with size I didn't mean the LOC in I can't provide an answer here, but I think reviewers should ask themselves the meta question (#35003 (comment)) and then hopefully there is rough consensus on the answer, whatever that will be. |
Early note: the majority of this PR consists of test coverage. The changes per se are quite small, just the issue shows up in multiple paths in slightly different forms.
The goal of the PR is to ensure I/O errors are not silently ignored during validation or P2P message handling. In some cases, such as
GETDATAorGETBLOCKTXNrequests, errors can be swallowed, leaving the node unresponsive to the request without any action. In a harder to reach but more severe case, a swallowed error can leave the node alive but stuck, unable to process new blocks and advance the chain. Also, silently ignoring errors obviously makes problems harder to diagnose. So this PR seeks to improve all of that.Currently, the same root cause, inability to access the blocks directory, can produce different behaviors depending on where it occurs:
If it happens during block connection, inside
AcceptBlock(), the error gets caught and triggers a fatal error graceful shutdown. This is the correct behavior.If it happens during block connection (due to the unguarded
FlatFileSeq::Open->fs::create_directoriescall), afterAcceptBlock(), insideActivateBestChain(), the error gets thrown and swallowed by the P2P general try-catch, leaving the node in a living but stuck state. WhereActivateBestChain()will always silently fail on any posterior block processing, retrying to active the failing block.If it happens during
GETDATArequest, the file system error gets swallowed by the message handling general try-catch, only logging a net-debug-level message. The remote peer is not disconnected, nor the node aborts. It just silently ignores the request. This is different to a missing block data, which currently logs the error + disconnects the peer.If it happens during
GETBLOCKTXNrequest, the file system error either gets swallowed by the message handling general try-catch or crashes the system via an assertion, depending on if the problem is at the blocks directory level or at the block data level, which is inconsistent.If it happens during
GETCFILTERSrequest, the file system error gets swallowed by the message handling general try-catch.If this happens in any of the index functions, the node crashes.
So, this PR makes failure handling consistent and ensures the node never enters a stuck state due to a block I/O error. Changes:
GETDATAnow treats the issue in the same way as other I/O errors: it logs the error + disconnect the remote peer.GETBLOCKTXNnow triggers a fatal error and graceful shutdown. The error here is stricter than inGETDATAbecause it can only access the last 10 blocks, which must be available for reorgs.ActivateBestChain()now triggers a fatal error and graceful shutdown when it happens. Fixing the silent stuck node state scenario.GETCFILTERSnow logs the error through the expected path: "Failed to find block filter in index". A more descriptive error message can be added in a follow-up PR.OpenFileto throw. Better error handling can be added in a follow-up PR.Testing Note
Can reproduce the different behaviors that cause the stuck node scenario by running the second commit’s unit test 28af683 or by manually introducing an exception in
ConnectTip()(which occurs insideActivateBestChain()). Previously, the thrown exception would have being caught by the general P2P try-catch and logged only at net-debug level, which most users do not have enabled, leaving the node alive but stuck, as subsequent block connections repeatedly attempt to process the failing block, throw the exception again, and get silently swallowed. Adding a functional test for this scenario is not possible due to the requirement of failing insideActivateBestChain(), which always occurs afterAcceptBlock(), which correctly captures the error.Separate Note
There is another assertion scenario that we can move to graceful shutdown (fatal error) in the compact block relay that I haven't done here.
Extra Note
If want to see a similar error producing a crash, but in the wallet, go to #34176.