Skip to content

[draft] Refactor some validation.cpp methods to return BlockValidationState#7

Closed
optout21 wants to merge 178 commits into
masterfrom
2605-validation-state-return
Closed

[draft] Refactor some validation.cpp methods to return BlockValidationState#7
optout21 wants to merge 178 commits into
masterfrom
2605-validation-state-return

Conversation

@optout21

@optout21 optout21 commented May 2, 2026

Copy link
Copy Markdown
Owner

Summary. Refactor validation result to be a return value instead of an output parameter in several validation.cpp methods.

Motivation. The benefits of the change are:

  • Exclude the potentially inconsistent case when the bool return value and the returned state are inconsistent
  • Exclude the ambiguity whether the passed in value of state is used or not (not obvious in chained calls)
  • Remove the possibility of unintuitive interaction between subsequent calls with the same state. In case of a validation error, a failure reason is set if the state was valid, but not if it was already invalid.
  • Slightly simpler: It's more evident which is the result; one less parameters.

This has grown out from bitcoin#33856, mentioned in comment here and here.

Details. Many methods follow the scheme where the validation state is returned in an output parameter (BlockValidationState& state), and and additional bool return value indicating success. In success case the convention is that state.IsValid() and the return value are both true. After the change there is only a BlockValidationState return value, which is either success (state.IsValid() == true), or an invalid/error case.

This change is a highly localized refactor, but touching a sensitive file.

Relevant methods called by ProcessNewBlockHeaders and AcceptBlock (directly and indirectly) are touched.

Changes are separated into commits by touched methods, ordered by bottom-to-top in the call hierarchy.

svanstaa and others added 7 commits April 8, 2026 12:16
add comment explaining coinstatsindex cache exclusion

update cache allocations to 10%/5%/5%
When setting the fd limit to unlimited, the node fails to start:

ulimit -n unlimited
build/bin/bitcoind
Error: Not enough file descriptors available. -1 available, 160 required.

This was caused by RaiseFileDescriptorLimit() casting limitFD.rlim_cur
to int, which for RLIM_INFINITY overflows to -1. Fix it by returning
std::numeric_limits<int>::max() instead.

This commit also adds a functional test, which is skipped on environments
with a hard limit below infinity.

Co-authored-by: Luke Dashjr <luke-jr+git@utopios.org>
Co-authored-by: winterrdog
When setting the fd limit to 1 >> 31, the node fails to start:

ulimit -n 214748364
build/bin/bitcoind
Error: Not enough file descriptors available. -2147483648 available, 160 required.

Similar to the previous commit, this is fixed by capping the limit
to std::numeric_limits<int>::max().

Co-authored-by: Luke Dashjr <luke-jr+git@utopios.org>
On 32-bit systems we build with _FILE_OFFSET_BITS=64 (see CMakeLists.txt),
which makes rlim_t 64-bit when building against glibc (see bits/resource.h).
Since size_t could be 32-bit, clamp RLIMIT_MEMLOCK to
std::numeric_limits<size_t>::max() in PosixLockedPageAllocator::GetLimit().

Co-authored-by: Luke Dashjr <luke-jr+git@utopios.org>
Cherry-pick include-what-you-use commit 52f85e1f4d99 onto the clang_22
branch tracked by ci/test/00_setup_env_native_iwyu.sh, fixing the false
positive where std::hash was mapped to <string_view>/<variant> instead of
its real provider (<functional>, <memory>, etc).
The default std::hash for shared_ptr compares by pointer.

CTransactionRefHash or a custom hasher should be used instead.

Co-authored-by: Hennadii Stepanov <32963518+hebasto@users.noreply.github.com>
LoadBlockIndex() adds to m_blocks_unlinked based only on nTx > 0, without
checking BLOCK_HAVE_DATA. Pruning preserves nTx but clears BLOCK_HAVE_DATA,
so a pruned block whose parent was header-only gets re-added on every
restart, causing the CheckBlockIndex() assertion that entries must have
data on disk to fail.

Check that BLOCK_HAVE_DATA is set before inserting into m_blocks_unlinked.

Fixes bitcoin#35050.
@optout21 optout21 force-pushed the 2605-validation-state-return branch 2 times, most recently from d78b817 to 44d3197 Compare May 2, 2026 12:48
svanstaa and others added 5 commits May 4, 2026 21:21
CHMAC_SHA256 and CHMAC_SHA512 leave two stack buffers populated on
return: rkey[] holds K' XOR ipad after the constructor, and temp[]
holds the inner-hash output after Finalize().

When the HMAC is keyed with sensitive material (chain code in
BIP32Hash() in hash.cpp for BIP32 child key derivation; PRK in
HKDF-Expand in hkdf_sha256_32.cpp, used for BIP324 transport keying),
rkey is one constant XOR from that key, and temp is a one-way digest
covering it.

Cleanse both buffers with memory_cleanse(), matching the convention
in chacha20.cpp and chacha20poly1305.cpp. No observable change for
callers.
A pruned stale-fork block whose parent doesn't have any transactions
shouldn't be added to m_blocks_unlinked when starting up a node.
HMAC primitives cleanse their internal stack buffers, but a caller's
ChainCode remains populated in memory after use. Promote ChainCode
from `typedef uint256` to a `base_blob<256>` subclass with a
memory_cleanse() destructor, so chain codes in CExtKey, CExtPubKey,
and local variables are cleansed on scope exit.

Retype MUSIG_CHAINCODE from `constexpr uint256` to `const ChainCode`
to match its BIP328 semantic role. Dropping `constexpr` (ChainCode is
no longer a literal type) also removes the GCC-14 consteval lambda
workaround.

Remove the duplicate typedef in pubkey.h (which includes hash.h
transitively). Two fuzz-test call sites in test/fuzz/key.cpp now
construct the chain-code argument explicitly rather than relying on
the typedef.
@optout21

Copy link
Copy Markdown
Owner Author

Note: bitcoin#33856 has been merged

@optout21 optout21 force-pushed the 2605-validation-state-return branch 2 times, most recently from aa67944 to 771a265 Compare May 20, 2026 08:42
MarcoFalke and others added 7 commits May 27, 2026 15:47
The variable is never set and will always be unbound.

The only way to set it correctly is via this hack:

.github/workflows/ci.yml-        # In the image build step, no external environment variables are available,
.github/workflows/ci.yml-        # so any settings will need to be written to the settings env file:
.github/workflows/ci.yml:        run: sed -i "s|\${INSTALL_BCC_TRACING_TOOLS}|true|g" ./ci/test/00_setup_env_native_asan.sh

So just silence the warning, which happens when running the task
locally:

```
./ci/test/00_setup_env_native_asan.sh: line 12: INSTALL_BCC_TRACING_TOOLS: unbound variable
```
Shellcheck marked the script as violating SC2044.

Instead of re-writing the Bash from scratch, just use Python.
build: exclude mptest target from compile commands
We are currently sometimes backlogged on waiting for runner selection.

Selecting Warp or GitHub-hosted runners directly from the repository context avoids serializing all CI jobs behind a metadata job. This keeps forks on public runners while allowing upstream jobs to schedule immediately on the intended runner labels.
The seed ID is calculated from a pubkey produced by treating the seed as
a private key. This calculation includes a pubkey compression parameter,
even thought that compression is completely irrelevant for the usage of
the seed as a BIP 32 seed. Thus migration should detect if a seed has
been used multiple times by checking if the computed master key was
already processed.

The spkm_migration fuzzer needs to have it's added descriptors
accounting to be updated for this fix.
Signed overflow on nScore updates is undefined behavior. Use
SaturatingAdd in AddLocal() and SeenLocal() so increments saturate at
INT_MAX instead of overflowing.

Add unit test coverage for saturation in both code paths.
Instead we fully reset CoinsResult-instances without forgetting any fields.

Fixes: bitcoin#35449

Variant resetting the other fields in Clear():
₿ hyperfine --warmup 2 "./build/bin/test_bitcoin -t coinselector_tests/knapsack_solver_test"
Benchmark 1: ./build/bin/test_bitcoin -t coinselector_tests/knapsack_solver_test
  Time (mean ± σ):      4.289 s ±  0.008 s    [User: 4.252 s, System: 0.027 s]
  Range (min … max):    4.279 s …  4.301 s    10 runs

This variant:
₿ hyperfine --warmup 2 "./build/bin/test_bitcoin -t coinselector_tests/knapsack_solver_test"
Benchmark 1: ./build/bin/test_bitcoin -t coinselector_tests/knapsack_solver_test
  Time (mean ± σ):      4.279 s ±  0.005 s    [User: 4.244 s, System: 0.026 s]
  Range (min … max):    4.271 s …  4.287 s    10 runs
…ngs.json

0654511 util: Check write failures before renaming settings.json (Shrey)

Pull request description:

  This PR Fixes bitcoin#35373.
  The error message was updated from: "This is probably caused by disk corruption or a crash" to: "This is probably caused by a full disk, disk corruption or a crash"

  The following files were updated:
  1. src/common/settings.cpp -- Updated the main string formatting for the parse failure.
  2. src/test/settings_tests.cpp -- Updated the exact string expectation in the C++ unit test.
  3. test/functional/feature_settings.py -- Updated the expected error message in the Python test suite.

  Configured the Bitcoin Core project natively on Windows using Visual Studio (cmake --preset vs2026-static -DBUILD_GUI=OFF) and it compiled with 100% of the C++ tests passed successfully (347 out of 347).

ACKs for top commit:
  maflcko:
    review ACK 0654511 💧
  winterrdog:
    ACK 0654511
  sedited:
    ACK 0654511
  ryanofsky:
    Code review ACK 0654511 with commit message and error message improved since last review

Tree-SHA512: d7b49860f2081a5a9d1d44917b0cf372a77f10cf21773c9b3291871c788a122ec5dde063fdf9e1052cf45b8d2667e15585b952e9f037e84f98fb41546bdd1fa9
@optout21 optout21 force-pushed the 2605-validation-state-return branch from ff0af9e to 811adb0 Compare June 19, 2026 14:29
@optout21 optout21 force-pushed the 2605-validation-state-return branch from 811adb0 to 6232800 Compare June 19, 2026 16:01
achow101 and others added 4 commits June 19, 2026 12:44
…nReserver reserve function

2818a17 test: add abortscan unit test (Pol Espinasa)
bc30e95 wallet: move fAbortRescan reset into WalletRescanReserver reserve() (Pol Espinasa)

Pull request description:

  Follow-up of bitcoin#35179
  For extra context refer to the conversations bitcoin#35179 (comment) and comments bellow it.

  Long story short: currently `ScanForWalletTransactions()` resets the value of `fAbortRescan` before starting the rescan loop. This can cause a race condition where some function (e.g. `importdescriptors`) starts a rescan and at the same time the user aborts it manually. Could happen that the `abortrescan` call returns True (success) but the rescan continues running as the value is overwritten.

  This PR fixes this by resetting the value of `fAbortRescan` at the very beginning, when the wallet rescan is reserved, removing the race condition. Also adds a test for it.

ACKs for top commit:
  achow101:
    ACK 2818a17
  w0xlt:
    ACK 2818a17
  nebula-21:
    ACK 2818a17
  pablomartin4btc:
    ACK 2818a17
  pinheadmz:
    ACK 2818a17

Tree-SHA512: 5b64b9a16a209dd145ccf7b2d7c3a7205038b3b214b02c3d9f1cbda75a6a1c9a41ad5c42cbeda43af6141793965613c47e44300d758149be9bfdf458e85a5520
1f3f0a4 Squashed 'src/secp256k1/' changes from 7262adb4b4..bd0287d650 (fanquake)

Pull request description:

  Includes:
  * bitcoin-core/secp256k1#1846
  * bitcoin-core/secp256k1#1848
  * bitcoin-core/secp256k1#1849
  * bitcoin-core/secp256k1#1851
  * bitcoin-core/secp256k1#1852
  * bitcoin-core/secp256k1#1854
  * bitcoin-core/secp256k1#1855
  * bitcoin-core/secp256k1#1859
  * bitcoin-core/secp256k1#1860
  * bitcoin-core/secp256k1#1861
  * bitcoin-core/secp256k1#1864
  * bitcoin-core/secp256k1#1865
  * bitcoin-core/secp256k1#1867

  The most interesting change to call out / that reviewers might want to test probably being bitcoin-core/secp256k1#1859.

ACKs for top commit:
  theStack:
    ACK 9caae50
  hebasto:
    ACK 9caae50.
  sedited:
    ACK 9caae50
  andrewtoth:
    ACK 9caae50

Tree-SHA512: 187e76d7f7ed603d81b7c355abfb184f37658fdb5109a2c78f72c615cdf862dfc9c3e2ff4556eb047d59348ba1a7030ca38342682c56dbfdb51610d2cc1de405
…sult fields

2447385 rpc: remove unused RPCResult::Type::ELISION (satsfy (Renato Britto))
7a85118 rpc: expand decodepsbt output script with explicit fields (satsfy (Renato Britto))
88e2a6a rpc: expand getaddressinfo embedded with explicit fields (Renato Britto)
a9f9e7d rpc: extract fee estimate result helpers (Renato Britto)
8a615a8 rpc: extract ListSinceBlockTxFields() helper (Renato Britto)
372ac28 rpc: extend TxDoc() for getblock verbosity 2/3 (Renato Britto)
0380a1c rpc: extend TxDoc() for getrawtransaction verbosity 2 (Renato Britto)
44fc3a2 rpc: introduce HelpElision variant and ElideGroup helper (Renato Britto)

Pull request description:

  Partially addresses bitcoin#29912. Motivated by bitcoin#34683, which exports OpenRPC from existing `RPCHelpMan` metadata. [Sample OpenRPC](https://gist.github.com/natobritto/8c4a1da04968d2325082ac4bca7d2408).

  Some RPC help definitions rely on `RPCResult::Type::ELISION` entries whose structure is only described in prose. This keeps human-readable help concise, but leaves parts of the result layout implicit and prevents tools from deriving complete machine-readable schemas from `RPCHelpMan` metadata.

  This PR replaces ELISION-based reuse with shared structured definitions, so result layouts are represented directly in metadata rather than only in text. At the same time, human-readable help remains compact via explicit help-rendering elision using `HelpElision`, so previously elided sections stay abbreviated without losing schema completeness.

  Affected RPCs: `getrawtransaction`, `getblock`, `listsinceblock`, `estimaterawfee`, `getaddressinfo`.

  RPC return values are unchanged. Human-readable help remains compact, while structured result metadata becomes explicit enough to derive complete machine-readable schemas.

  A related `RPCResult::Type::ELISION` use in `importdescriptors` was split out into the follow-up PR bitcoin#34867 because it changes the generated help output, per [this review comment](bitcoin#34764 (comment)).

  Changes:
  - Introduce `HelpElision` (`NONE`, `START`, `SKIP`) and `ElideGroup()`, replacing the tri-state `print_elision`
  - Add an `RPCResult` copy-with-replacement-options constructor to support applying elision while keeping `m_opts` const
  - Extend `TxDoc()` / `TxDocOptions` to support reusable transaction layouts with optional `prevout`, `fee`, `hex`, and elision behavior
  - Replace ELISION-based reuse in `getrawtransaction` and `getblock` with explicit structured definitions
  - Factor shared result layouts into `GetBlockFields()`, `ListSinceBlockTxFields()`, `FeeRateBucketDoc()`, `GetAddressInfoEmbeddedFields()` and `FeeEstimateHorizonDoc()`
  - Expand `listsinceblock.removed`, `estimaterawfee` horizons/buckets and `getaddressinfo.embedded` into explicit metadata while preserving concise help output

ACKs for top commit:
  achow101:
    ACK 2447385
  w0xlt:
    reACK 2447385
  janb84:
    re ACK 2447385
  willcl-ark:
    ACK 2447385

Tree-SHA512: 8dc03c45c388ebdb4f8f1613af2576fc127a8d5425efe518cd0d0ed2439a38b2ed1236413471672c85f52ee22c4a17677c18fd4689bc6570496fc3af4cd4112f
735b255 support: clamp RLIMIT_MEMLOCK to size_t (Sjors Provoost)
8ab4b9f init: clamp fd limits to int (Sjors Provoost)
4afbabd Fix startup failure with RLIM_INFINITY fd limits (Sjors Provoost)

Pull request description:

  When setting the fd limit to unlimited, the node fails to start:

  ```sh
  ulimit -n unlimited
  build/bin/bitcoind
  Error: Not enough file descriptors available. -1 available, 160 required.
  ```

  This was caused by `RaiseFileDescriptorLimit()` (introduced in bitcoin#2568) casting `limitFD.rlim_cur` to `int`, which for `RLIM_INFINITY` overflows to `-1`. Fix it by returning `std::numeric_limits<int>::max()` instead.

  Some platforms implement `RLIM_INFINITY` as the maximum uint64, others as int64 (-1). So simply changing the return type to `uint64_t` wouldn't work.

  Similarly, though unlikely to actually happen:

  ```sh
  ulimit -n 214748364
  build/bin/bitcoind
  Error: Not enough file descriptors available. -2147483648 available, 160 required.
  ```

  The second commit expands the fix by clamping all values above `std::numeric_limits<int>::max()` instead of letting them overflow.

  This PR also expands `test/functional/feature_init.py` to cover these, using `resource.setrlimit`. The check is skipped on environments with a hard limit below infinity (or that don't have the Python [Resource module](https://docs.python.org/3/library/resource.html)).

  macOS by default has a hard limit of infinity, but on e.g. Ubuntu the default hard limit is 524288.

  The third commit applies a similar fix to `PosixLockedPageAllocator::GetLimit()` for 32-bit systems, but without a test.

ACKs for top commit:
  winterrdog:
    Re-ACK 735b255
  achow101:
    ACK 735b255
  sedited:
    Re-ACK 735b255
  pinheadmz:
    ACK 735b255

Tree-SHA512: 0ce0292ecd61456bdec6943b06cbb9ecfc5180ee6dce850f8496ef54af22c1fae6ea473085202f5ba6f72e4dc51a29247620c9a0eae31e96658adc77b293129f
@optout21 optout21 force-pushed the 2605-validation-state-return branch 4 times, most recently from 7ad0822 to 12a93b8 Compare June 20, 2026 06:08
optout21 and others added 14 commits June 20, 2026 08:18
Return BlockValidationState by value instead of using an out-parameter,
similar to the TestBlockValidity refactoring in 74690f4.

Remove redundant int return from btck_chainstate_manager_process_block_header.
Previously returned both an int result and an output validation state parameter, creating ambiguity
where non-zero could mean either invalid header or processing failure. Since ProcessNewBlockHeaders already provides complete validation info, the int return was redundant.

Co-authored-by: stringintech <stringintech@gmail.com>
Co-authored-by: stickies-v <stickies-v@protonmail.com>
In `ChainstateManager::AcceptBlock()`, don't ignore the return value from
`FlushStateToDisk`.
@optout21 optout21 force-pushed the 2605-validation-state-return branch from 12a93b8 to cc4bac4 Compare June 20, 2026 06:21
@optout21

optout21 commented Jun 20, 2026

Copy link
Copy Markdown
Owner Author

--> #18 , #19

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.