index: make txospender reorg recovery crash-safe#224
Draft
l0rinc wants to merge 22 commits into
Draft
Conversation
Cursor iteration is only supported by the coins database view. Pass `CCoinsViewDB` pointers to UTXO stats code and use the concrete DB pointer in the coins_view fuzz target before removing the abstract hook. Co-authored-by: sedited <seb.kung@gmail.com>
`CCoinsView` does not need to expose cursor iteration now that the few cursor users take `CCoinsViewDB` directly. Keep `Cursor()` on `CCoinsViewDB`, where iteration is supported, and remove the empty, forwarding, and throwing base-view overrides. This also removes the coins_view fuzz target probe that only asserted the unsupported `CCoinsViewCache::Cursor()` throw path. Co-authored-by: sedited <seb.kung@gmail.com>
The UTXO stats helpers require a non-null coins DB view after cursor users were narrowed to `CCoinsViewDB`. Use references for the DB view in the UTXO stats helpers and for local DB-view/block-manager variables at call sites, so the path no longer spells a nullable contract. Co-authored-by: sedited <seb.kung@gmail.com>
`CCoinsViewDB::Cursor()` now owns cursor iteration and constructs the cursor directly. Drop the leftover handling for the old generic nullable cursor contract from DB cursor call sites. Co-authored-by: sedited <seb.kung@gmail.com>
Prepare for letting explicitly configured local addresses bypass the reachable-net filter without tying that behavior to LOCAL_MANUAL score. Pass an `add_even_if_unreachable` bool through `AddLocal()`, defaulting to `false`.
When -onlynet excludes a network, AddLocal() rejects local addresses from that network. This also rejects addresses explicitly configured via -externalip, so a configuration like -onlynet=ipv4 -externalip=<onion> never advertises the onion address. Set the AddLocal() unreachable-net override only when adding -externalip addresses so the bypass remains limited to explicit user configuration.
Add unit coverage for the -onlynet/-externalip interaction. Check that an unreachable address still fails with normal AddLocal() arguments, while the same address succeeds when the unreachable-net override is set for explicit local-address configuration.
Extend p2p_addr_selfannouncement to restart the node with -onlynet=ipv4 -externalip=<onion> and verify the onion address appears in localaddresses despite its network being unreachable.
ChooseSelectionResult computes the bump-fee discount as: summed_bump_fees - combined_bump_fee Where summed_bump_fees is the sum of per-UTXO ancestor bump fees and combined_bump_fee is the true combined cost taking into account shared ancestors. Both variables use creates a fresh MiniMiner snapshot of the mempool. Because of that the two snapshots of the mempool might be different. An artificial feerate decrease of an ancestor using prioritizesettransaction can make combined_bump_fee > summed_bump_fees. This cause calling bumpfeediscount with a negative vaule triggering an assertion >= 0. This commit fixes this by only calling bumpfeediscount when the discount is strictly positive. Co-authored-by: dergoegge <n.goeggi@gmail.com>
2b49b34 to
dfa04f1
Compare
Remove the `Mutex` from the `coins_view` and `coinscache_sim` pool startup helpers. Fuzz targets are entered sequentially within a process and parallel fuzzing uses separate processes/forks, which each have their own copy of the global thread pool. Therefore, a mutex to prevent two in-process callers from racing to start the pool isn't needed.
240d5f7 fuzz: Drop unnecessary mutexes (marcofleon) Pull request description: Quick cleanup addressing bitcoin#35295 (comment). This follows the same logic as 48df093 from bitcoin#35521. ACKs for top commit: maflcko: lgtm ACK 240d5f7 nervana21: ACK 240d5f7 l0rinc: code review ACK 240d5f7 Tree-SHA512: 869127dc8cec4fc0688f92d67dc22f643cf3741c62d645eb4a959a9d8f46590efe19765f7df97ee36f9a3ca759c50fe983a3e6655e40a53ea99904e7a1ec00f6
… values 3ae3a94 wallet: avoid call bumpfeediscount with negative values (Pol Espinasa) Pull request description: in bitcoin#34232 dergoegge reported an assertion fail in `SetBumpFeeDiscount`. **Context** The bump-fee discount in the savings that we can have when multiple UTXOs that we selected for our transaction share a common unconfirmed ancestor transaction. We know we can have some savings because we first calculate the `summed_bump_fees` and then compare to the `combined_bump_fee`. For context: - `bump_fees`: Extra fees that the new transaction must pay to contribute to get his ancestor confirmed. - `summed_bump_fees`: The sum of each input ancestor `bump_fee`. If we have two inputs with unconfirmed ancestors A and B and both have a `bump_fee = 100` then `summed_bump_fees = 200`. - `combined_bump_fee`: Is the total `bump_fees` summed of all inputs, but taking into account shared ancestors. If A and B share the transaction ancestor then `combined_bump_fee = 100` not `200` as the transaction must be bumped only once. If `summed_bumpfees` > `combined_bump_fee` we are overestimating the `bump_fee` as we are counting multiple times the same ancestors so we can discount it using `SetBumpFeeDiscount(summed_bump_fees - combined_bump_fees)`. **Problem** To calculate `summed_bump_fees` and `combined_bump_fee` we use two different fresh MiniMiner snapshots of the mempool. Because they are called in different moments the two snapshots of the mempool might be different. An artificial feerate decrease of an ancestor using `prioritizesettransaction` can make `combined_bump_fee > summed_bump_fees` creating a negative discount. This cause calling `SetBumpFeeDiscount` with a negative vaule triggering an assertion `discount >= 0`. **Fix** This PR fixes it by ensuring not only that a discount exist but also that is greater the 0. **Test** It is hard to manually trigger this race condition. dergoegge coded a patch and test to trigger it that can be used to test the fix. polespinasa@5320e2f ACKs for top commit: achow101: ACK 3ae3a94 pablomartin4btc: ACK 3ae3a94 Tree-SHA512: e76693eb66c4883ed3ef5edf1baa972657b0d532487803706312d486b4d4a4997f5dcbc64781a1a21b7b5628b06fcc97a6633382509061734d0f5615e851bef1
72db4ac coins: drop stale cursor null checks (Lőrinc) 3d2f2d8 coins: pass UTXO stats view by reference (Lőrinc) 35aedb2 coins: drop cursor from base view (Lőrinc) c6fbe2f coins: pass DB view to cursor users (Lőrinc) Pull request description: **Problem:** `CCoinsView::Cursor()` makes cursor iteration look like a generic coins view operation, but cursor iteration is only supported by the DB-backed coins view. The cache override only threw, and the `coins_view` fuzz target only asserted that deterministic unsupported throw path. **Fix:** Make cursor iteration a `CCoinsViewDB` operation. Cursor users now take the DB-backed view directly, `CCoinsView` no longer exposes `Cursor()`, and the fuzz target keeps DB-backed cursor coverage while dropping the unsupported cache throw probe. The UTXO stats path is also tightened to pass the non-null DB view by reference, and stale null handling for DB cursors is removed. This was extracted from review discussion in bitcoin#35295 (comment) and extended based on bitcoin#35562 (comment). ACKs for top commit: achow101: ACK 72db4ac sedited: Re-ACK 72db4ac w0xlt: ACK 72db4ac andrewtoth: ACK 72db4ac Tree-SHA512: 12a81330a6ec1b91a7e4393f3761ea9ed4702ecb24312f1defa5a9a079a396ce921fc52f74fe296e5ac7ab20d5b5a8a84e858c96847f333c58b7fa9de9e8143e
…rate response 4c9de7d external_signer: validate fingerprint from enumerate response (Kyle 🐆) Pull request description: `enumeratesigners` takes the `fingerprint` field from the external signer's `enumerate` output and stores it without checking it. That value is later handed back to the signer command as `--fingerprint <value>` (e.g. in `displayaddress`), so a malformed value propagates unchecked. A master key fingerprint is 4 bytes, i.e. 8 hex characters. This adds a check that the reported fingerprint is exactly 8 hex characters and throws a clear error otherwise. A functional test covers empty, wrong-length, and non-hex inputs. ACKs for top commit: Sjors: utACK 4c9de7d achow101: ACK 4c9de7d sedited: ACK 4c9de7d Tree-SHA512: 7c3303b24e234e13a4c20c0b93552145b9ccffc29d1bae42ce8a2faf548377f051e52f8ffb3924679065b27d15fc7bf3859e5ae32a2bb185738cc29bc0ade486
dab7f2c test: cover -externalip/onlynet interaction in functional test (will) 657a5aa test: cover -externalip bypassing -onlynet (will) 8c87e32 net: let -externalip bypass -onlynet (will) f4af02e net: add an add_even_if_unreachable argument to AddLocal (will) Pull request description: `-onlynet` is documented to restrict automatic outbound connections, but it also currently prevents `-externalip` addresses from being advertised when their network is not in the `-onlynet` set. This happens because `AddLocal()` rejects addresses outside `g_reachable_nets`, regardless of whether the address was explicitly configured by the user. Previous attempts to fix this (bitcoin#24835 and bitcoin#25690) removed the `g_reachable_nets` check from `AddLocal()`. This PR instead adds an explicit `add_even_if_unreachable` argument to `AddLocal()`. The argument defaults to `false`, and is set to `true` only when adding addresses from `-externalip`. As a result, explicitly configured `-externalip` addresses can still be advertised even when their network is excluded by `-onlynet`, while discovered, mapped, bound, Tor control, and I2P SAM addresses continue to use the existing reachable-network filter. This keeps the fix scoped to `-externalip` and addresses the concern raised in bitcoin#25690: > I think it might also be weird for a user to activate -onlynet and keep on advertising their clearnet address to the network The branch adds unit coverage for `AddLocal()` and functional coverage in `p2p_addr_selfannouncement.py` for `-onlynet=ipv4 -externalip=<onion>`. Fixes: bitcoin#25336 Fixes: bitcoin#25669 ACKs for top commit: achow101: ACK dab7f2c mzumsande: re-ACK dab7f2c w0xlt: ACK dab7f2c Tree-SHA512: a4ac9334b85da8b6902d3850e21d3a1c9d7dce70bcb79182448c8d5684e24462cd6e440385af7aa4420d9582e4dff9dc9e827ca7a6da0363fff2d3c531784d9b
`BuildChainTestingSetup` duplicates block construction on a chosen parent. Move the overload to `TestChain100Setup`, disable mempool selection because callers supply the complete transaction list, and regenerate commitments after changing the coinbase.
Persist spender A, reorg without flushing to competing spender B, then restore durable chainstate A and reopen the index. Current eager reorg cleanup deletes A's record while B's stale record remains. The TODO assertions record that behavior for the following fix.
Txospender records are written before the best-block locator is committed. Deleting them on disconnect creates the inverse crash window: rollback can restore an active spend whose record is gone. Keep records append-only and make FindSpender() return only exact spends from active blocks. Disconnect data is no longer needed, so stop requesting it.
`WriteSpenderInfos()` has one caller after disconnect cleanup is removed. Inline it into `CustomAppend()` and keep its database-only details out of the header.
Corrupt inactive spender B's transaction bytes while later active spender A remains readable. Current lookup learns block activity only after reading the transaction, so B's I/O error prevents it from reaching A. The TODO assertion records that behavior for the following fix.
Append-only keys identify only disk positions, so an inactive record cannot be rejected until its transaction is read. Missing unflushed block data can therefore make a stale record hide a later active spend. Store one position-to-hash mapping per block in the same batch as its spender records. Lookup can then reject inactive blocks before transaction I/O. Existing databases have no mappings. Continue past unreadable legacy candidates and return the saved error only when no active spender is found. Refs bitcoin#24539
dfa04f1 to
0bcd884
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem:
TxoSpenderIndexwrites spender records before its best-block locator is durable.After an interrupted reorg, disconnect cleanup can remove the restored active spend while a competing record remains, so restart returns a spend from an inactive block.
Because records identify only disk positions, unreadable data for that stale block can also prevent a later active spend from being found.
Fix: Keep spender records append-only and filter them by active-chain membership during lookup.
Store one block-position-to-hash mapping per block in the same batch as its spender records, allowing inactive blocks to be rejected before transaction I/O.
For legacy records without mappings, continue past unreadable candidates and report the saved I/O error only if no active spend is found.
Reproducer: This matures a coinbase, kills the node after its spend is indexed but before the index locator becomes durable, then restarts and queries the original output.
This exercises immediate process-crash recovery, not power-loss durability.
Stale spend after an interrupted block connection
The spend never enters the mempool, so the pre-fix result comes from the stale index record.
Before: The result includes a
spendingtxidandblockhashfrom the crashed block.After: The result contains only the queried
txidandvout.History
TxoSpenderIndexwrites can outlive the durable index locator and that rejecting unreadable stale records requires block identity. This PR adds one block-position mapping per block.