Skip to content

validation: improve block data I/O error handling in P2P paths#35003

Open
furszy wants to merge 5 commits into
bitcoin:masterfrom
furszy:2026_abc_io_exception
Open

validation: improve block data I/O error handling in P2P paths#35003
furszy wants to merge 5 commits into
bitcoin:masterfrom
furszy:2026_abc_io_exception

Conversation

@furszy

@furszy furszy commented Apr 4, 2026

Copy link
Copy Markdown
Member

Early note: the majority of this PR consists of test coverage. The changes per se are quite small, just the issue shows up in multiple paths in slightly different forms.

The goal of the PR is to ensure I/O errors are not silently ignored during validation or P2P message handling. In some cases, such as GETDATA or GETBLOCKTXN requests, errors can be swallowed, leaving the node unresponsive to the request without any action. In a harder to reach but more severe case, a swallowed error can leave the node alive but stuck, unable to process new blocks and advance the chain. Also, silently ignoring errors obviously makes problems harder to diagnose. So this PR seeks to improve all of that.

Currently, the same root cause, inability to access the blocks directory, can produce different behaviors depending on where it occurs:


  1. If it happens during block connection, inside AcceptBlock(), the error gets caught and triggers a fatal error graceful shutdown. This is the correct behavior.

  2. If it happens during block connection (due to the unguarded FlatFileSeq::Open -> fs::create_directories call), after AcceptBlock(), inside ActivateBestChain(), the error gets thrown and swallowed by the P2P general try-catch, leaving the node in a living but stuck state. Where ActivateBestChain() will always silently fail on any posterior block processing, retrying to active the failing block.

  3. If it happens during GETDATA request, the file system error gets swallowed by the message handling general try-catch, only logging a net-debug-level message.
The remote peer is not disconnected, nor the node aborts. It just silently ignores the request. This is different to a missing block data, which currently logs the error + disconnects the peer.

  4. If it happens during GETBLOCKTXN request, the file system error either gets swallowed by the message handling general try-catch or crashes the system via an assertion, depending on if the problem is at the blocks directory level or at the block data level, which is inconsistent.

  5. If it happens during GETCFILTERS request, the file system error gets swallowed by the message handling general try-catch.

  6. If this happens in any of the index functions, the node crashes.

So, this PR makes failure handling consistent and ensures the node never enters a stuck state due to a block I/O error. Changes:

  1. GETDATA now treats the issue in the same way as other I/O errors: it logs the error + disconnect the remote peer.
  2. GETBLOCKTXN now triggers a fatal error and graceful shutdown. The error here is stricter than in GETDATA because it can only access the last 10 blocks, which must be available for reorgs.
  3. ActivateBestChain() now triggers a fatal error and graceful shutdown when it happens. Fixing the silent stuck node state scenario.
  4. GETCFILTERS now logs the error through the expected path: "Failed to find block filter in index". A more descriptive error message can be added in a follow-up PR.
  5. The index-related functions now ignore the error instead of crashing the node. This matches how the code was written, as these paths were not expecting OpenFile to throw. Better error handling can be added in a follow-up PR.

Testing Note
Can reproduce the different behaviors that cause the stuck node scenario by running the second commit’s unit test 28af683 or by manually introducing an exception in ConnectTip() (which occurs inside ActivateBestChain()). Previously, the thrown exception would have being caught by the general P2P try-catch and logged only at net-debug level, which most users do not have enabled, leaving the node alive but stuck, as subsequent block connections repeatedly attempt to process the failing block, throw the exception again, and get silently swallowed. Adding a functional test for this scenario is not possible due to the requirement of failing inside ActivateBestChain(), which always occurs after AcceptBlock(), which correctly captures the error.

Separate Note
There is another assertion scenario that we can move to graceful shutdown (fatal error) in the compact block relay that I haven't done here.

Extra Note
If want to see a similar error producing a crash, but in the wallet, go to #34176.

@DrahtBot

DrahtBot commented Apr 4, 2026

Copy link
Copy Markdown
Contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/35003.

Reviews

See the guideline and AI policy for information on the review process.

Type Reviewers
ACK l0rinc
Concept ACK pinheadmz
Approach NACK josibake
Stale ACK frankomosh, sedited, rkrux, w0xlt
User requested bot ignore maflcko

If your review is incorrectly listed, please copy-paste <!--meta-tag:bot-skip--> into the comment that the bot should ignore.

Conflicts

Reviewers, this pull request conflicts with the following ones:

  • #35731 (Indexes: Harden the flush-error notification invariant by arejula27)
  • #35676 (util: Abort in CheckDiskSpace/FlatFileSeq::Open on rare exceptions by maflcko)
  • #35139 (test: Add thread-safe fast-failing test macros by maflcko)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

LLM Linter (✨ experimental)

Possible typos and grammar issues:

  • txospenderindex -> txspendingindex [misspelled index name in the docstring; the intended term is unclear otherwise]

Possible places where named args for integral literals may be used (e.g. func(x, /*named_arg=*/0) in C++, and func(x, named_arg=0) in Python):

  • FlatFileSeq(data_dir, "a", 100) in src/test/flatfile_tests.cpp

Possible places where comparison-specific test macros should replace generic comparisons:

  • src/test/flatfile_tests.cpp BOOST_CHECK_THROW(seq.Allocate(FlatFilePos(0, 0), 1, out_of_space), fs::filesystem_error); -> consider using BOOST_CHECK_EXCEPTION with a predicate that checks the expected error message/details instead of only the exception type.

2026-07-02 15:13:56

@pinheadmz

Copy link
Copy Markdown
Member

concept ACK
Thanks for the thorough PR description! Definitely helps me to understand the code changes before even looking at them. Can also see its minimal changes with lots of test coverage, will review.

@frankomosh frankomosh left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tACK 72f7a67

Built from source. All new tests pass:

  • flatfile_tests: 9 cases, no errors
  • validation_chainstate_tests: 5 cases, no errors
  • blockmanager_tests: 8 cases, no errors
  • p2p_handle_io_errors.py: both scenarios pass

Also manually reverted the GETBLOCKTXN fix (commit ba034f4) by removing the fatalError call and replacing with a silent return. The functional test correctly caught this kind of mutation. the test expected the node to shut down, but the mutant node stayed alive, causing a timeout failure. Mutant killed by p2p_handle_io_errors.py. Unit tests did not detect this mutation. Also process_messages and process_message fuzz harnesses both passed without detecting the mutation, confirming the functional test is providing the critical oracle here.

Line 5915 in SendMessages has assert(ret) after a ReadBlock call in the compact block announcement path. I think this is the same pattern as the GETBLOCKTXN assert fixed in commit 4. I believe this is the "another assertion scenario in compact block relay" mentioned in the PR description. Is this planned as a follow-up? Happy to take it if so.

@DrahtBot DrahtBot requested a review from pinheadmz April 8, 2026 04:45

@maflcko maflcko left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

concept ack, left a test suggestion, which could make review easier

Comment thread test/functional/p2p_handle_io_errors.py
@furszy

furszy commented Apr 11, 2026

Copy link
Copy Markdown
Member Author

Line 5915 in SendMessages has assert(ret) after a ReadBlock call in the compact block announcement path. I think this is the same pattern as the GETBLOCKTXN assert fixed in commit 4. I believe this is the "another assertion scenario in compact block relay" mentioned in the PR description. Is this planned as a follow-up? Happy to take it if so.

Yes @frankomosh. All yours :).

@furszy furszy force-pushed the 2026_abc_io_exception branch from 5a25f94 to fc86ae4 Compare April 11, 2026 22:23
Comment thread test/functional/p2p_handle_io_errors.py Outdated
block_hash = node.getblockhash(3) # Not in recent cache
with simulate_io_error(node.blocks_path):
# Bug: we currently swallow the GETDATA request, never reply to the other party, and log issue under debug-NET
with node.assert_debug_log(expected_msgs=["[ProcessMessages] [net] ProcessMessages(getdata, 37 bytes): Exception 'filesystem error"],

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: intentionally not checking for a specific filesystem error msg, since it slightly vary across OS. Specific msg doesn't matter, this first commit just proves the error is swallowed in master.

@furszy furszy force-pushed the 2026_abc_io_exception branch from fc86ae4 to 825dfc2 Compare April 12, 2026 12:09

@maflcko maflcko left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like the test fails on the second commit, or so?

Also, provided some more test nits. Feel free to ignore.

825dfc2~4 📭

Show signature

Signature:

untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
trusted comment: 825dfc200520d503fec64346ed01a8b12ec8fa54~4 📭
HuyWCmP9dlHdaEd4PL2XkITSC+f4T6nxsowBmRtzkzVkthldhUIEhYd8dWE2Z+BGoWVYLPWjRRxIPvoLUhqCCg==

Comment thread test/functional/p2p_handle_io_errors.py Outdated
Comment thread test/functional/p2p_handle_io_errors.py Outdated
self.setup_clean_chain = True

def test_block_connect_on_broken_fs(self):
self.log.info("Test block connect on inaccessible filesystem")

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit in 75a8a95:

From the pull description:

If it happens during block connection, after AcceptBlock(), inside ActivateBestChain(), the file system error gets thrown and swallowed by the P2P general try-catch, leaving the node in a living but stuck state. Where ActivateBestChain() will always silently fail on any posterior block processing, retrying to active the failing block.

You claim that a failure in ActivateBestChain will leave the node alive, but stuck. However, when writing a test for this, I found that it properly shut down:

diff --git a/test/functional/p2p_handle_io_errors.py b/test/functional/p2p_handle_io_errors.py
index 4438ad1dd5..4706dfebd4 100755
--- a/test/functional/p2p_handle_io_errors.py
+++ b/test/functional/p2p_handle_io_errors.py
@@ -14,2 +14,3 @@ from test_framework.blocktools import (
     create_block,
+    create_empty_fork,
 )
@@ -49,5 +50,6 @@ class P2PBlockIOErrorTest(BitcoinTestFramework):
         self.setup_clean_chain = True
+        self.extra_args = [['-fastprune']]
 
-    def test_block_connect_on_broken_fs(self):
-        self.log.info("Test block connect on inaccessible filesystem")
+    def test_block_accept_on_broken_fs(self):
+        self.log.info("Test block accept on inaccessible filesystem")
         node = self.nodes[0]
@@ -70,2 +72,24 @@ class P2PBlockIOErrorTest(BitcoinTestFramework):
 
+    def test_block_connect_on_broken_fs(self):
+        self.log.info("Test block connect on inaccessible filesystem")
+        node = self.nodes[0]
+        peer = node.add_p2p_connection(P2PInterface())
+        blocks = create_empty_fork(node, fork_length=2)
+        large_block = self.generatetodescriptor(node, 1, f"raw({'55'*100_000})")[0]
+        peer.send_and_ping(msg_block(blocks[0]))
+        (node.blocks_path / 'blk00002.dat').unlink()
+        node.getblock(large_block)
+
+        with node.assert_debug_log(expected_msgs=["ActivateBestChain failed (Failed to read block.)"]):
+            peer.send_without_ping(msg_block(blocks[1]))
+            peer.wait_for_disconnect()
+
+        node.wait_until_stopped(
+            expect_error=True,
+            expected_stderr=re.compile(r"fatal internal error"),
+        )
+
+        # Restart node for next test
+        self.start_node(0, extra_args=['-reindex'])
+
     def test_getdata_on_broken_fs(self):
@@ -125,2 +149,3 @@ class P2PBlockIOErrorTest(BitcoinTestFramework):
 
+        self.test_block_accept_on_broken_fs()
         self.test_block_connect_on_broken_fs()

I guess it could make sense to add the test?

Also, it could make sense to clarify that point 2 in the pull request description refers to a thrown exception, presumably only from create_directories, and does not refer to any IO error?

Just a nit, and up to you, if you want to keep this pull focussed only on the create_directories call, or be a more general io error handling improvement in P2P paths, with test coverage.

@furszy furszy Apr 13, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You claim that a failure in ActivateBestChain will leave the node alive, but stuck. However, when writing a test for this, I found that it properly shut down.

Yeah ok. Your test exercises a missing block file, not the inability to access the directory. If we want to replicate the stuck, the goal there should be to get the node stuck by making it throw inside the create_directories call.

I had a test early in this branch that reproduced the "throw inside ActivateBestChain" through the P2P path in a deterministic way (which is not easy due to AcceptBlock happening first). But I did not like that it relied on another bug in reconsiderblock to trigger this one.

Basically, the approach was to invalidate the tip, submit a fork, make the directory inaccessible, and then reconsider the block, which throws internally but resets the block validity flags (the operation is not atomically reverted upon failure). After that, sending any number of descendant blocks via P2P will be swallowed due to the inability to activate the very first block. But it was ugly to depend on reconsiderblock to trigger it.

I guess it could make sense to add the test?

Yes! for sure. Pulling it.

Also, it could make sense to clarify that point 2 in the pull request description refers to a thrown exception, presumably only from create_directories, and does not refer to any IO error?

Sure. Will do.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had a test early in this branch that reproduced the "throw inside ActivateBestChain" through the P2P path in a deterministic way (which is not easy due to AcceptBlock happening first). But I did not like that it relied on another bug in reconsiderblock to trigger this one.

Basically, the approach was to invalidate the tip, submit a fork, make the directory inaccessible, and then reconsider the block, which throws internally but resets the block validity flags (the operation is not atomically reverted upon failure). After that, sending any number of descendant blocks via P2P will be swallowed due to the inability to activate the very first block. But it was ugly to depend on reconsiderblock to trigger it.

Ah interesting. I presume the bug still exists after this pull requests, as the ReconsiderBlock function is not changed here? Just wondering, as the bug seems unrelated to the bug fixed in this pull request. Also, I agree that such a test may be too spicy to include here.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I presume the bug still exists after this pull requests, as the ReconsiderBlock function is not changed here?

Would say it is partially fixed but haven't tried it. ReconsiderBlock internally calls ActivateBestChain, which triggers a fatal error after this PR, instead of throwing an exception. So it is a matter of whether we are flushing the block flags reset to disk prior to aborting or not. Because, if we do, the node would fail early on any following up startup.

Comment thread test/functional/p2p_handle_io_errors.py Outdated
Comment thread test/functional/p2p_handle_io_errors.py Outdated
@furszy

furszy commented Apr 13, 2026

Copy link
Copy Markdown
Member Author

Looks like the test fails on the second commit, or so?

It seems the assertion error message is libc-dependent.
Linux logs "Assertion `expr' failed" while MacOS logs "Assertion failed: (expr), ...".

Will make the wait_until_stopped expected error a bit more general and check for "Assertion" wording only. That is more than enough to check the crash reason.
The last commit makes it clear with the fatal error change anyway.

@furszy furszy force-pushed the 2026_abc_io_exception branch from 825dfc2 to 476fc0f Compare April 13, 2026 16:10
@maflcko

maflcko commented Jun 25, 2026

Copy link
Copy Markdown
Member

Haven't read all of the discussion threads, but since my last review, the only change seems to be moving the chmod, which doesn't seem like a good change. (See inline comment)

Also, the suggestion by @josibake to use exceptions here sounds interesting. However, I haven't looked at the details and code, so it is harder to tell which is the right fit here. I'd be happy to review either approach, and wonder what other reviewers prefer here?

lgtm ACK d4f9539 🐻

Show signature

Signature:

untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
trusted comment: lgtm ACK d4f95399b5af4643b90bb396c4d172e908ae87c2 🐻
oH+HVBgFxqvZd9iiXWWh2X30aRqlpbsZRmLXAWKmHc3lkuFDKeMRDVJRktd+GT8xFT0DR5CtMzWrWtrUHDDUCA==

@DrahtBot DrahtBot requested review from josibake, l0rinc and w0xlt June 25, 2026 05:56

@maflcko maflcko left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.

Comment thread test/functional/p2p_handle_io_errors.py Outdated
old_mode = stat.S_IMODE(os.stat(parent_dir).st_mode)

os.rename(blocks_path, blocks_bak)
os.chmod(parent_dir, 0o500) # Prevent re-creation of blocks/

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#35003 (comment): I don't think this change was a beneficial change. When the chmod fails, the test should also fail, not silently pass and even skip running the fn/yield.

I think the correct fix would be to revert this change and then wait for the test to fail, and then figure out if that env/platform should be skipped or supported.

@furszy furszy force-pushed the 2026_abc_io_exception branch from d4f9539 to 81cddf0 Compare June 30, 2026 20:21
@furszy

furszy commented Jun 30, 2026

Copy link
Copy Markdown
Member Author

I hopefully tackled all remaining feedback. Also added coverage for the GETCFILTERS p2p path (swallowed exception) and the RPC getrawtransaction and gettxspendingprevout paths (low-level filesystem exception reaching the user).

@l0rinc l0rinc left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

code review ACK 81cddf0

The changes look correct to me and the new coverage exercises the relevant failure paths well.
Seems to me the remaining compact-block announcement should likely get the same fatal-error treatment, but I'm also fine with handling it as a follow-up.
(nit: PR description needs slight adjustment now regarding commit order)

Comment thread src/net_processing.cpp
// pruned after we release cs_main above, so this read should never fail.
assert(ret);
// Unless an I/O error occurs, in which case we want to shut down gracefully.
if (!ret) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems to me compact-block announcements still abort on the same kind of failure:

bitcoin/src/net_processing.cpp

Lines 5997 to 5998 in dc282ff

const bool ret{m_chainman.m_blockman.ReadBlock(block, *pBestIndex)};
assert(ret);

Is that deliberate, or could we use the same fatal-error style there?

nit: please see my suggestion above on how we can simplify the code after the assertion removal.

@DrahtBot DrahtBot requested a review from maflcko June 30, 2026 22:08
@furszy

furszy commented Jul 1, 2026

Copy link
Copy Markdown
Member Author

It seems to me compact-block announcements still abort on the same kind of failure:

bitcoin/src/net_processing.cpp

Lines 5997 to 5998 in dc282ff

const bool ret{m_chainman.m_blockman.ReadBlock(block, *pBestIndex)};
assert(ret);

Is that deliberate, or could we use the same fatal-error style there?

Yes, it is in the PR description. You probably forgot this one #35003 (review) and my reply #35003 (comment).

@maflcko maflcko left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

only test changes since last review, re-ACK 81cddf0 🌊

Show signature

Signature:

untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
trusted comment: only test changes since last review, re-ACK 81cddf0d878a27b84c85db831be0165648b89e94 🌊
25kQW99wcgHBdEbUId8fCqwmWawZrgdJHnNKS3hwKbmceslmlwVE98zG84KPEXu7zjtO4F8GL1Yeaj9ewBhNAQ==

Comment thread test/functional/p2p_handle_io_errors.py Outdated

@contextlib.contextmanager
def simulate_io_error(target_path):
# Make files under target_path inaccessible by renaming its parent dir,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in cfdcd3b: I don't think this recent force push is correct.

Not the target_path's parent is renamed, but the target_path itself.

@furszy furszy Jul 2, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ups, comment fixed.

node.getblockcount()

def test_getcfilters_on_broken_fs(self):
"""GETCFILTERS must log the filter-index read failure and keep running."""

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in the first commit: Just like getdata, it would be kind to the peer to disconnect it, rather than silently not reply to the request?

@furszy furszy Jul 2, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking about tackling it in a separate PR because it would be good to cover all filter-related messages at once, not just this one. We disconnect inside PrepareBlockFilterRequest for all of them, but not in any of the subsequent lookups.

furszy and others added 5 commits July 2, 2026 10:58
Add tests showing that block disk I/O failures triggered
through P2P requests are silently swallowed.

The idea is to simulate I/O errors by making the blocks directory
temporarily inaccessible while the node is running. This allows block
reads to fail when serving P2P requests.

Aside from the well behaving test cases, this covers a few cases
where the node mishandles the error:

- GETDATA: failures should be handled locally by disconnecting the
  peer and logging the error, without impacting node operation.

- GETBLOCKTXN: failures when accessing recent blocks violate our
  "last 10 blocks must always be available for reorgs" rule and
  must result in a fatal error, rather than being caught and only
  logged by the P2P message handling general try-catch.

- txindex/txospenderindex: getrawtransaction and gettxspendingprevout
  open the block file directly, so filesysteme exceptions leak out as
  a generic RPC error instead of being handled cleanly.

- GETCFILTERS: failures should be logged through the filter index read
  path and leave the node running, rather than being swallowed by the
  P2P messages general try-catch.

Co-authored-by: MarcoFalke <*~=`'#}+{/-|&$^_@721217.xyz>
The callers of this function don't expect it to throw.

For example, when the blocks directory becomes inaccessible
(e.g. volume detach or permission changed), FlatFileSeq::Open's
call to fs::create_directories() throws filesystem_error. On the
block connection path, if this occurs during ActivateBestChain(),
just after AcceptBlock() the exception bypasses FatalError entirely.
It gets caught by ProcessMessage's general catch-all with only a
NET-level debug log, swallowing the error msg for any regular
user, and every subsequent block arrival retries the same failing
read, leaving the node looking healthy while the block processing
mechanism is stuck.
Cover the two ActivateBestChain paths that read data from
disk: ConnectTip (block connection) and DisconnectTip (reorg).

Simulate inaccessible block and undo files, ensure fatal error
and shutdown is triggered.
Verifies that BlockManager behaves the same way whether a block file is
missing or the blocks directory is inaccessible.

Reads (ReadBlock, ReadRawBlock, ReadBlockUndo) return false, writes
(WriteBlock, WriteBlockUndo) return null/false, and failures are logged.

The goal is consistent failure handling. Previously, missing block files
returned false and logged an error, while an inaccessible blocks directory
was throwing an exception.
…tdown

A failure here indicates a serious inconsistency or disk issue, and the
current behavior is to abort immediately, which is correct but not the best.
Replace assert with a fatal error so the node shuts down through the normal
shutdown path instead.

This does not change the outcome (the node still stops), but allows a more
controlled teardown, giving other components a chance to flush their state
even when a particular block (or the blocks dir) on disk is unreachable.
E.g. the blocks dir might be the only affected, the mempool might be ok,
and we don't want to lose such information.
@furszy furszy force-pushed the 2026_abc_io_exception branch from 81cddf0 to 7680140 Compare July 2, 2026 15:13
@l0rinc

l0rinc commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

reACK 7680140

@DrahtBot DrahtBot requested a review from maflcko July 2, 2026 15:58
@maflcko

maflcko commented Jul 2, 2026

Copy link
Copy Markdown
Member

test-doc-only re-ACK 7680140 🍏

Show signature

Signature:

untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
trusted comment: test-doc-only re-ACK 768014068b8ccdb205b4ad8e666eed07db247221  🍏
eTqlGYFVii78opeRBwUyzB67YFv2RzmwLLXHJWOpZnMwKSqaIkoE59cn2wV8OTyGy5qmBLInA6fVRY01kz3RAQ==

@sedited

sedited commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

@rkrux @w0xlt @frankomosh @josibake can you take another look here?

@josibake

josibake commented Jul 9, 2026

Copy link
Copy Markdown
Member

Approach NACK

While I certainly appreciate the work @furszy has put into this, the PR in its current form is not the direction I think we should be going. I spent some time reading and thinking through a more comprehensive error handling strategy for the codebase in order to convince myself I'm not pushing back simply for aesthetics, and I am convinced removing exceptions from the low level file primitive and folding it into a nullptr is the wrong direction.

I would much prefer something like this in, instead of the FlatFileSeq commit:

index b060108c18..df89984d53 100644
--- a/src/node/blockstorage.cpp
+++ b/src/node/blockstorage.cpp
@@ -821,13 +821,23 @@ void BlockManager::UnlinkPrunedFiles(const std::set<int>& setFilesToPrune) const

 AutoFile BlockManager::OpenBlockFile(const FlatFilePos& pos, bool fReadOnly) const
 {
-    return AutoFile{m_block_file_seq.Open(pos, fReadOnly), m_obfuscation};
+    try {
+        return AutoFile{m_block_file_seq.Open(pos, fReadOnly), m_obfuscation};
+    } catch (const fs::filesystem_error& e) {
+        LogError("OpenBlockFile failed for %s: %s", pos.ToString(), e.what());
+        return AutoFile{nullptr, m_obfuscation};
+    }
 }

 /** Open an undo file (rev?????.dat) */
 AutoFile BlockManager::OpenUndoFile(const FlatFilePos& pos, bool fReadOnly) const
 {
-    return AutoFile{m_undo_file_seq.Open(pos, fReadOnly), m_obfuscation};
+    try {
+        return AutoFile{m_undo_file_seq.Open(pos, fReadOnly), m_obfuscation};
+    } catch (const fs::filesystem_error& e) {
+        LogError("OpenUndoFile failed for %s: %s", pos.ToString(), e.what());
+        return AutoFile{nullptr, m_obfuscation};
+    }
 }

 fs::path BlockManager::GetBlockPosFilename(const FlatFilePos& pos) const

This preserves the exception, and moves the translation of that error into BlockManager which I have argued is the appropriate layer for handling the exception. This directly solves the stated problem: exceptions were being swallowed at a higher layer and not handled. It solves the problem by handling the exception at the appropriate layer, without needing to touch the lower level file primitives.

What my proposal does not address is the BlockFilterIndex uses of FlatFileSeq. I think thats fine, as those are not in the critical path. I'd argue its even desirable, as we may want to have a different policy on how FlatFileSeq failures are handled there, further demonstrating the strength of this approach.

I do understand that the approach in this PR was done with consistency in mind, but I don't think that is enough of a justification when the thing being made consistent is, as I've tried to argue here, a bad design.

Comment thread src/flatfile.cpp
fs::create_directories(path.parent_path());
const fs::path path = FileName(pos);
// Create parent path only if we are going to write.
if (!read_only) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: this is unrelated to error handling afaict and could be pulled out into its own PR with its own test, no?

@maflcko

maflcko commented Jul 9, 2026

Copy link
Copy Markdown
Member

Let's recall this pull has more than 150 comments and is sitting for 3 months (which is reasonable for its size). However, it doesn't seem to be getting traction and has outstanding issues, some of which are enough to NACK it.

So, I don't think the pull will be merged as-is. Even if it was merged, there'd be a bunch of follow-ups, so it would be good to decide how to address them before merge.

I think it would be good to step back and answer a meta question:

  • Do we even care about providing fallbacks for those exceptions? Let's recall they can only happen when there is a coding bug, hardware corruption, or the users physically fiddles with the hardware (e.g. disconnect an external blocksdir)

There are different answers:

  • "They are so rare, we don't care about them at all". Then I think this pull (and other ones) can be closed and we can all move on to other things.
  • "They are rare, but providing a fatal shutdown path in all of them is too tedious". Then, I think it is fine to manually terminate/abort in the functions (See util: Abort in CheckDiskSpace/FlatFileSeq::Open on rare exceptions #35676)
  • "They are rare, but providing a fatal shutdown path in all of them should be done". Then, I think there needs to be a single large pull request to do it in all paths.
  • ... Some other answer I am not seeing?

@josibake

josibake commented Jul 9, 2026

Copy link
Copy Markdown
Member

Thanks for zooming out @maflcko , I think its helpful. I'll respond here, but this is starting to feel like a discussion that can be moved to an issue around error handling in general, considering the discussion now spans two PRs.

Do we even care about providing fallbacks for those exceptions?

This is a policy question. I think its up to the caller to decide, which is why I've been pushing back on folding multiple errors into a nullptr, and pushing back on having the low level function abort instead of propagating the error up. Said differently, the answer to this question will change in the future, and it will likely be different for different callers.

Some other answer I am not seeing?

I think we should stop swallowing errors with broad catch alls that do nothing. Instead, if we can't catch it, let it terminate. If we would prefer a graceful fatal shutdown instead, have broad catch all that executes a safe shut down. Even better, have each boundary own their own error handling policy as I recommend here: BlockManager should own storage errors, BlockfilterIndex should manage storage errors related to indexes, etc etc.

In each of these boundaries, if they have the information they need to know that a graceful fatal shutdown is possible/safe/etc, they can do that. If the cant make that call, they should allow the error to propagate.

@DrahtBot DrahtBot requested a review from josibake July 9, 2026 14:23
@furszy

furszy commented Jul 9, 2026

Copy link
Copy Markdown
Member Author

Restating the discussion we already had: #35003 (comment).

Today, every filesystem error in FlatFileSeq::Open is handled through the same path except for this one case. Your proposal @josibake keeps that one case special, how leaving a function with two error-signaling paths for the same class of failure makes sense to you? That is just error-prone. It doesn't really matter which kind of error we are talking about.

Overall, following the by the books approach without pondering the actual sources is just bad in my view. And treating this bug fix PR as a broader architectural design discussion spot, instead of just creating a new issue or PR (as it was suggested in my first replies to you), is just blocking progress unnecessarily.

Let's recall this pull has more than 150 comments and is sitting for 3 months (which is reasonable for its size)

The PR is 97% about adding tests. It changes 5 lines of code in the node at most.

@maflcko

maflcko commented Jul 9, 2026

Copy link
Copy Markdown
Member

Let's recall this pull has more than 150 comments and is sitting for 3 months (which is reasonable for its size)

The PR is 97% about adding tests. It changes 5 lines of code in the node at most.

Sorry for being unclear, with size I didn't mean the LOC in ./src, but the review size, also approximated by the number of diverse tests. Let's recall that the exceptions involved here concern all threads: scheduler, indexes, initload, main, p2p, rpc, except for the script and fetcher threads. Also, the exceptions here (or the return value) can propagate up ~anywhere in the codebase (except maybe for wallet and unrelated low-level utils). So any reviewer must be familiar with the whole codebase.

I can't provide an answer here, but I think reviewers should ask themselves the meta question (#35003 (comment)) and then hopefully there is rough consensus on the answer, whatever that will be.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

10 participants