Skip to content

index: make txospender reorg recovery crash-safe#224

Draft
l0rinc wants to merge 22 commits into
masterfrom
l0rinc/txospender-crash-recovery
Draft

index: make txospender reorg recovery crash-safe#224
l0rinc wants to merge 22 commits into
masterfrom
l0rinc/txospender-crash-recovery

Conversation

@l0rinc

@l0rinc l0rinc commented Jul 11, 2026

Copy link
Copy Markdown
Owner

Problem: TxoSpenderIndex writes spender records before its best-block locator is durable.
After an interrupted reorg, disconnect cleanup can remove the restored active spend while a competing record remains, so restart returns a spend from an inactive block.
Because records identify only disk positions, unreadable data for that stale block can also prevent a later active spend from being found.

Fix: Keep spender records append-only and filter them by active-chain membership during lookup.
Store one block-position-to-hash mapping per block in the same batch as its spender records, allowing inactive blocks to be rejected before transaction I/O.
For legacy records without mappings, continue past unreadable candidates and report the saved I/O error only if no active spend is found.

Reproducer: This matures a coinbase, kills the node after its spend is indexed but before the index locator becomes durable, then restarts and queries the original output.
This exercises immediate process-crash recovery, not power-loss durability.

Stale spend after an interrupted block connection
D="$(mktemp -d)"
cli() { build/bin/bitcoin-cli -regtest "-datadir=$D" "$@"; }
start() { build/bin/bitcoind -regtest "-datadir=$D" -txospenderindex=1 >/dev/null 2>&1 & p=$!; until cli getblockcount >/dev/null 2>&1; do sleep .1; done; }
stop() { cli stop >/dev/null; wait "$p"; }
synced() { until cli getindexinfo | grep -q '"synced": true'; do sleep .1; done; }
# Persist a spendable coinbase and its index state
    ninja -C build >/dev/null && start && \
    cli generatetodescriptor 101 'raw(51)' >/dev/null && \
    coin="$(cli getblock "$(cli getblockhash 1)" 2 | awk -F '"' '/"txid":/ { print $4; exit }')" && \
    synced && stop
# Crash after the spender record is written, before its locator is durable
    start && synced && \
    tx="$(cli createrawtransaction "[{\"txid\":\"$coin\",\"vout\":0}]" '[{"data":"00"}]')" && \
    cli generateblock 'raw(51)' "[\"$tx\"]" >/dev/null && \
    cli syncwithvalidationinterfacequeue && kill -9 "$p"; wait "$p" || true
# Query the stale record after restart
    start && synced && \
    cli gettxspendingprevout "[{\"txid\":\"$coin\",\"vout\":0}]" '{"mempool_only":false}' && stop

The spend never enters the mempool, so the pre-fix result comes from the stale index record.

Before: The result includes a spendingtxid and blockhash from the crashed block.

After: The result contains only the queried txid and vout.

History
  • #24539 added disconnect-time record removal to avoid returning stale reorg spends. Crash rollback exposes the inverse failure: removal can lose a restored active spend, so this PR preserves records and checks active-chain membership during lookup.
  • #34897 identified that eager TxoSpenderIndex writes can outlive the durable index locator and that rejecting unreadable stale records requires block identity. This PR adds one block-position mapping per block.

l0rinc and others added 10 commits June 24, 2026 14:30
Cursor iteration is only supported by the coins database view.

Pass `CCoinsViewDB` pointers to UTXO stats code and use the concrete DB pointer in the coins_view fuzz target before removing the abstract hook.

Co-authored-by: sedited <seb.kung@gmail.com>
`CCoinsView` does not need to expose cursor iteration now that the few cursor users take `CCoinsViewDB` directly.
Keep `Cursor()` on `CCoinsViewDB`, where iteration is supported, and remove the empty, forwarding, and throwing base-view overrides.

This also removes the coins_view fuzz target probe that only asserted the unsupported `CCoinsViewCache::Cursor()` throw path.

Co-authored-by: sedited <seb.kung@gmail.com>
The UTXO stats helpers require a non-null coins DB view after cursor users were narrowed to `CCoinsViewDB`.

Use references for the DB view in the UTXO stats helpers and for local DB-view/block-manager variables at call sites, so the path no longer spells a nullable contract.

Co-authored-by: sedited <seb.kung@gmail.com>
`CCoinsViewDB::Cursor()` now owns cursor iteration and constructs the cursor directly.

Drop the leftover handling for the old generic nullable cursor contract from DB cursor call sites.

Co-authored-by: sedited <seb.kung@gmail.com>
Prepare for letting explicitly configured local addresses bypass the
reachable-net filter without tying that behavior to LOCAL_MANUAL score.

Pass an `add_even_if_unreachable` bool through `AddLocal()`, defaulting
to `false`.
When -onlynet excludes a network, AddLocal() rejects local addresses
from that network. This also rejects addresses explicitly configured via
-externalip, so a configuration like -onlynet=ipv4 -externalip=<onion>
never advertises the onion address.

Set the AddLocal() unreachable-net override only when adding -externalip
addresses so the bypass remains limited to explicit user configuration.
Add unit coverage for the -onlynet/-externalip interaction.

Check that an unreachable address still fails with normal AddLocal()
arguments, while the same address succeeds when the unreachable-net
override is set for explicit local-address configuration.
Extend p2p_addr_selfannouncement to restart the node with -onlynet=ipv4
-externalip=<onion> and verify the onion address appears in
localaddresses despite its network being unreachable.
ChooseSelectionResult computes the bump-fee discount as: summed_bump_fees - combined_bump_fee
Where summed_bump_fees is the sum of per-UTXO ancestor bump fees and combined_bump_fee is the
true combined cost taking into account shared ancestors.

Both variables use creates a fresh MiniMiner snapshot of the mempool. Because of that
the two snapshots of the mempool might be different. An artificial feerate decrease
of an ancestor using prioritizesettransaction can make combined_bump_fee > summed_bump_fees.
This cause calling bumpfeediscount with a negative vaule triggering an assertion >= 0.

This commit fixes this by only calling bumpfeediscount when the discount is strictly positive.

Co-authored-by: dergoegge <n.goeggi@gmail.com>
@l0rinc l0rinc force-pushed the l0rinc/txospender-crash-recovery branch from 2b49b34 to dfa04f1 Compare July 11, 2026 16:28
@l0rinc l0rinc changed the title index: recover txospender records after crashes index: make txospender reorg recovery crash-safe Jul 14, 2026
marcofleon and others added 12 commits July 14, 2026 19:21
Remove the `Mutex` from the `coins_view` and `coinscache_sim`
pool startup helpers. Fuzz targets are entered sequentially within a
process and parallel fuzzing uses separate processes/forks, which each
have their own copy of the global thread pool. Therefore, a mutex to
prevent two in-process callers from racing to start the pool isn't needed.
240d5f7 fuzz: Drop unnecessary mutexes (marcofleon)

Pull request description:

  Quick cleanup addressing bitcoin#35295 (comment).

  This follows the same logic as 48df093 from bitcoin#35521.

ACKs for top commit:
  maflcko:
    lgtm ACK 240d5f7
  nervana21:
    ACK 240d5f7
  l0rinc:
    code review ACK 240d5f7

Tree-SHA512: 869127dc8cec4fc0688f92d67dc22f643cf3741c62d645eb4a959a9d8f46590efe19765f7df97ee36f9a3ca759c50fe983a3e6655e40a53ea99904e7a1ec00f6
… values

3ae3a94 wallet: avoid call bumpfeediscount with negative values (Pol Espinasa)

Pull request description:

  in bitcoin#34232 dergoegge reported an assertion fail in `SetBumpFeeDiscount`.

  **Context**

  The bump-fee discount in the savings that we can have when multiple UTXOs that we selected for our transaction share a common unconfirmed ancestor transaction.

  We know we can have some savings because we first calculate the `summed_bump_fees` and then compare to the `combined_bump_fee`.
  For context:
  - `bump_fees`: Extra fees that the new transaction must pay to contribute to get his ancestor confirmed.
  - `summed_bump_fees`: The sum of each input ancestor `bump_fee`. If we have two inputs with unconfirmed ancestors A and B and both have a `bump_fee = 100` then `summed_bump_fees = 200`.
  - `combined_bump_fee`: Is the total `bump_fees` summed of all inputs, but taking into account shared ancestors. If A and B share the transaction ancestor then `combined_bump_fee = 100` not `200` as the transaction must be bumped only once.

  If `summed_bumpfees` > `combined_bump_fee` we are overestimating the `bump_fee` as we are counting multiple times the same ancestors so we can discount it using `SetBumpFeeDiscount(summed_bump_fees - combined_bump_fees)`.

  **Problem**

  To calculate `summed_bump_fees` and `combined_bump_fee` we use two different fresh MiniMiner snapshots of the mempool. Because they are called in different moments the two snapshots of the mempool might be different. An artificial feerate decrease of an ancestor using `prioritizesettransaction` can make `combined_bump_fee > summed_bump_fees` creating a negative discount. This cause calling `SetBumpFeeDiscount` with a negative vaule triggering an assertion `discount >= 0`.

  **Fix**

  This PR fixes it by ensuring not only that a discount exist but also that is greater the 0.

  **Test**

  It is hard to manually trigger this race condition. dergoegge coded a patch and test to trigger it that can be used to test the fix.
  polespinasa@5320e2f

ACKs for top commit:
  achow101:
    ACK 3ae3a94
  pablomartin4btc:
    ACK 3ae3a94

Tree-SHA512: e76693eb66c4883ed3ef5edf1baa972657b0d532487803706312d486b4d4a4997f5dcbc64781a1a21b7b5628b06fcc97a6633382509061734d0f5615e851bef1
72db4ac coins: drop stale cursor null checks (Lőrinc)
3d2f2d8 coins: pass UTXO stats view by reference (Lőrinc)
35aedb2 coins: drop cursor from base view (Lőrinc)
c6fbe2f coins: pass DB view to cursor users (Lőrinc)

Pull request description:

  **Problem:** `CCoinsView::Cursor()` makes cursor iteration look like a generic coins view operation, but cursor iteration is only supported by the DB-backed coins view.
  The cache override only threw, and the `coins_view` fuzz target only asserted that deterministic unsupported throw path.

  **Fix:** Make cursor iteration a `CCoinsViewDB` operation.
  Cursor users now take the DB-backed view directly, `CCoinsView` no longer exposes `Cursor()`, and the fuzz target keeps DB-backed cursor coverage while dropping the unsupported cache throw probe.
  The UTXO stats path is also tightened to pass the non-null DB view by reference, and stale null handling for DB cursors is removed.

  This was extracted from review discussion in bitcoin#35295 (comment) and extended based on bitcoin#35562 (comment).

ACKs for top commit:
  achow101:
    ACK 72db4ac
  sedited:
    Re-ACK 72db4ac
  w0xlt:
    ACK 72db4ac
  andrewtoth:
    ACK 72db4ac

Tree-SHA512: 12a81330a6ec1b91a7e4393f3761ea9ed4702ecb24312f1defa5a9a079a396ce921fc52f74fe296e5ac7ab20d5b5a8a84e858c96847f333c58b7fa9de9e8143e
…rate response

4c9de7d external_signer: validate fingerprint from enumerate response (Kyle 🐆)

Pull request description:

  `enumeratesigners` takes the `fingerprint` field from the external signer's `enumerate` output and stores it without checking it. That value is later handed back to the signer command as `--fingerprint <value>` (e.g. in `displayaddress`), so a malformed value propagates unchecked.

  A master key fingerprint is 4 bytes, i.e. 8 hex characters. This adds a check that the reported fingerprint is exactly 8 hex characters and throws a clear error otherwise. A functional test covers empty, wrong-length, and non-hex inputs.

ACKs for top commit:
  Sjors:
    utACK 4c9de7d
  achow101:
    ACK 4c9de7d
  sedited:
    ACK 4c9de7d

Tree-SHA512: 7c3303b24e234e13a4c20c0b93552145b9ccffc29d1bae42ce8a2faf548377f051e52f8ffb3924679065b27d15fc7bf3859e5ae32a2bb185738cc29bc0ade486
dab7f2c test: cover -externalip/onlynet interaction in functional test (will)
657a5aa test: cover -externalip bypassing -onlynet (will)
8c87e32 net: let -externalip bypass -onlynet (will)
f4af02e net: add an add_even_if_unreachable argument to AddLocal (will)

Pull request description:

  `-onlynet` is documented to restrict automatic outbound connections, but it also currently prevents `-externalip` addresses from being advertised when their network is not in the `-onlynet` set. This happens because `AddLocal()` rejects addresses outside `g_reachable_nets`, regardless of whether the address was explicitly configured by the user.

  Previous attempts to fix this (bitcoin#24835 and bitcoin#25690) removed the `g_reachable_nets` check from `AddLocal()`.

  This PR instead adds an explicit `add_even_if_unreachable` argument to `AddLocal()`. The argument defaults to `false`, and is set to `true` only when adding addresses from `-externalip`.

  As a result, explicitly configured `-externalip` addresses can still be advertised even when their network is excluded by `-onlynet`, while discovered, mapped, bound, Tor control, and I2P SAM addresses continue to use the existing reachable-network filter.

  This keeps the fix scoped to `-externalip` and addresses the concern raised in bitcoin#25690:

  > I think it might also be weird for a user to activate -onlynet and keep on advertising their clearnet address to the network

  The branch adds unit coverage for `AddLocal()` and functional coverage in `p2p_addr_selfannouncement.py` for `-onlynet=ipv4 -externalip=<onion>`.

  Fixes: bitcoin#25336
  Fixes: bitcoin#25669

ACKs for top commit:
  achow101:
    ACK dab7f2c
  mzumsande:
    re-ACK dab7f2c
  w0xlt:
    ACK dab7f2c

Tree-SHA512: a4ac9334b85da8b6902d3850e21d3a1c9d7dce70bcb79182448c8d5684e24462cd6e440385af7aa4420d9582e4dff9dc9e827ca7a6da0363fff2d3c531784d9b
`BuildChainTestingSetup` duplicates block construction on a chosen parent.

Move the overload to `TestChain100Setup`, disable mempool selection because callers supply the complete transaction list, and regenerate commitments after changing the coinbase.
Persist spender A, reorg without flushing to competing spender B, then restore durable chainstate A and reopen the index.

Current eager reorg cleanup deletes A's record while B's stale record remains. The TODO assertions record that behavior for the following fix.
Txospender records are written before the best-block locator is committed. Deleting them on disconnect creates the inverse crash window: rollback can restore an active spend whose record is gone.

Keep records append-only and make FindSpender() return only exact spends from active blocks. Disconnect data is no longer needed, so stop requesting it.
`WriteSpenderInfos()` has one caller after disconnect cleanup is removed. Inline it into `CustomAppend()` and keep its database-only details out of the header.
Corrupt inactive spender B's transaction bytes while later active spender A remains readable.

Current lookup learns block activity only after reading the transaction, so B's I/O error prevents it from reaching A. The TODO assertion records that behavior for the following fix.
Append-only keys identify only disk positions, so an inactive record cannot be rejected until its transaction is read. Missing unflushed block data can therefore make a stale record hide a later active spend.

Store one position-to-hash mapping per block in the same batch as its spender records. Lookup can then reject inactive blocks before transaction I/O.

Existing databases have no mappings. Continue past unreadable legacy candidates and return the saved error only when no active spender is found.

Refs bitcoin#24539
@l0rinc l0rinc force-pushed the l0rinc/txospender-crash-recovery branch from dfa04f1 to 0bcd884 Compare July 15, 2026 00:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants