Skip to content

txospenderindex: use zero-byte entry values#35634

Merged
sedited merged 2 commits into
bitcoin:masterfrom
l0rinc:l0rinc/txospenderindex-empty-values
Jul 4, 2026
Merged

txospenderindex: use zero-byte entry values#35634
sedited merged 2 commits into
bitcoin:masterfrom
l0rinc:l0rinc/txospenderindex-empty-values

Conversation

@l0rinc

@l0rinc l0rinc commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Problem: TxoSpenderIndex values are never read: FindSpender() only uses the key, which already encodes the spender's disk position, but the values are written as single-byte \0(instead of empty values).

Fix: Write zero-byte markers for new txospenderindex entries instead.
Existing indexes stay readable because both formats use the same keys and the value is ignored.
Rebuilding is only needed to shrink old entries.

Related work: This complements #35568, which saves another ~4.5 GB by dropping bloom filters.
A #35531-style rewrite could let existing indexes reclaim both savings without a full rebuild.

Reproducer: Automated tests weren't added, but the manual reproducer below builds both commits, checks old/new format compatibility through gettxspendingprevout, and prints rebuilt index sizes.
On my mainnet datadir this PR shrank txospenderindex by ~3.4 GB (from 89G to 86G).

Script + sample output
BEFORE="89b4000ae06c72c5a14ee05ad70d3aece3f1b382" AFTER="1be799ef858c33158a302857b084e169b5bf8c1f" DATA_DIR="/mnt/my_storage/BitcoinData" LOG="${DATA_DIR}/debug.log" OUT='[{"txid":"0437cd7f8525ceed2324359c2d0ba26006d92d856a9c20fa0241106ee5a597c9","vout":0}]' SPEND="f4184fc596403b9d638783cf57adfe4c75c605f6356fbc91338530e9831e9e16"; \
for b in before after; do [ -x ./build-$b/bin/bitcoin-cli ] && ./build-$b/bin/bitcoin-cli -datadir="${DATA_DIR}" stop >/dev/null 2>&1 || true; done; sleep 10; \
git reset --hard >/dev/null 2>&1 && git clean -fxd >/dev/null 2>&1 && (git fetch origin "$BEFORE" "$AFTER" >/dev/null 2>&1 || true) && \
for c in before:$BEFORE after:$AFTER; do git checkout ${c#*:} >/dev/null 2>&1 && cmake -B build-${c%:*} -DCMAKE_BUILD_TYPE=Release >/dev/null 2>&1 && cmake --build build-${c%:*} -j --target bitcoind bitcoin-cli >/dev/null 2>&1; done && \
for b in before after; do ./build-$b/bin/bitcoin-cli -datadir="${DATA_DIR}" stop >/dev/null 2>&1 || true; done; sleep 10; \
start_node() { [ "$2" = wipe ] && rm -rf "${DATA_DIR}/indexes/txospenderindex" "${LOG}"; : > "${LOG}"; ./build-"$1"/bin/bitcoind -datadir="${DATA_DIR}" -txospenderindex=1 -connect=0 -printtoconsole=0 & pid=$!; while ! grep -Fq 'txospenderindex is enabled at height' "${LOG}" 2>/dev/null; do kill -0 "$pid" 2>/dev/null || { tail -100 "${LOG}"; return 1; }; sleep 5; done; }; \
check_spend() { result="$(./build-"$1"/bin/bitcoin-cli -datadir="${DATA_DIR}" gettxspendingprevout "$OUT" '{"mempool_only":false}')"; echo "$2: $result"; echo "$result" | grep -q "$SPEND"; }; \
stop_node() { ./build-"$1"/bin/bitcoin-cli -datadir="${DATA_DIR}" stop >/dev/null; wait "$pid"; }; \
echo "prevout: 0437cd7f8525ceed2324359c2d0ba26006d92d856a9c20fa0241106ee5a597c9:0 -> ${SPEND}" && \
start_node before wipe && check_spend before "old binary reads old-format index" && du -sh "${DATA_DIR}/indexes/txospenderindex"; stop_node before && \
start_node after && check_spend after "new binary reads old-format index"; stop_node after && \
start_node after wipe && check_spend after "new binary reads new-format index" && du -sh "${DATA_DIR}/indexes/txospenderindex"; stop_node after && \
start_node before && check_spend before "old binary reads new-format index"; stop_node before

Expected result, trimmed to the relevant compatibility and size lines:

prevout: 0437cd7f8525ceed2324359c2d0ba26006d92d856a9c20fa0241106ee5a597c9:0 -> f4184fc596403b9d638783cf57adfe4c75c605f6356fbc91338530e9831e9e16
old binary reads old-format index: [ ... "spendingtxid": "f4184fc596403b9d638783cf57adfe4c75c605f6356fbc91338530e9831e9e16" ... ]
89G     /mnt/my_storage/BitcoinData/indexes/txospenderindex
new binary reads old-format index: [ ... "spendingtxid": "f4184fc596403b9d638783cf57adfe4c75c605f6356fbc91338530e9831e9e16" ... ]
new binary reads new-format index: [ ... "spendingtxid": "f4184fc596403b9d638783cf57adfe4c75c605f6356fbc91338530e9831e9e16" ... ]
86G     /mnt/my_storage/BitcoinData/indexes/txospenderindex
old binary reads new-format index: [ ... "spendingtxid": "f4184fc596403b9d638783cf57adfe4c75c605f6356fbc91338530e9831e9e16" ... ]

@DrahtBot

DrahtBot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/35634.

Reviews

See the guideline for information on the review process.

Type Reviewers
ACK sedited, ekzyis, andrewtoth, davidgumberg

If your review is incorrectly listed, please copy-paste <!--meta-tag:bot-skip--> into the comment that the bot should ignore.

Comment thread src/index/txospenderindex.cpp
@l0rinc l0rinc force-pushed the l0rinc/txospenderindex-empty-values branch from a8f57da to 1be799e Compare July 1, 2026 20:36
@DrahtBot

DrahtBot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

🚧 At least one of the CI tasks failed.
Task iwyu: https://github.com/bitcoin/bitcoin/actions/runs/28546146458/job/84631833696
LLM reason (✨ experimental): CI failed because the IWYU (include-what-you-use) check detected missing/incorrect header includes and intentionally exited with “Failure generated from IWYU”.

Hints

Try to run the tests locally, according to the documentation. However, a CI failure may still
happen due to a number of reasons, for example:

  • Possibly due to a silent merge conflict (the changes in this pull request being
    incompatible with the current code in the target branch). If so, make sure to rebase on the latest
    commit of the target branch.

  • A sanitizer issue, which can only be found by compiling with the sanitizer and running the
    affected test.

  • An intermittent issue.

Leave a comment here, if you need help tracking down a confusing failure.

l0rinc added 2 commits July 1, 2026 14:46
`TxoSpenderIndex` uses LevelDB values only as presence markers. `FindSpender` iterates over keys and loads the spending transaction from the disk position stored in each key.
Write new entries with a zero-byte span instead of a serialized empty string, so rebuilt indexes avoid one extra byte per spender entry.
Existing indexes remain readable because the lookup path does not deserialize the marker value.
Document that newly indexed `txospenderindex` entries use less disk space and that existing indexes remain readable.
Users only need to rebuild the index if they want previously indexed entries rewritten with the smaller marker.
@l0rinc l0rinc force-pushed the l0rinc/txospenderindex-empty-values branch from 1be799e to 1134022 Compare July 1, 2026 21:46
@DrahtBot DrahtBot removed the CI failed label Jul 1, 2026

@sedited sedited left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK 1134022

@sedited sedited requested a review from fjahr July 2, 2026 07:48

@ekzyis ekzyis left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

utACK 1134022

Problem: TxoSpenderIndex values are never read: FindSpender() only uses the key, which already encodes the spender's disk position, but the values are written as single-byte \0 values.

Nit: As mentioned in the release notes, this is about saving one byte, but the problem description sounds like there's a bug.

@andrewtoth andrewtoth left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK 1134022

@davidgumberg

Copy link
Copy Markdown
Contributor

crACK 1134022

@sedited sedited merged commit b393985 into bitcoin:master Jul 4, 2026
27 checks passed
@fjahr

fjahr commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

post merge ACK 1134022

@l0rinc l0rinc deleted the l0rinc/txospenderindex-empty-values branch July 4, 2026 14:35
oleonardolima added a commit to oleonardolima/rust-bitcoinkernel that referenced this pull request Jul 9, 2026
…05fc7ab

81405fc7ab Merge bitcoin/bitcoin#35689: test: Inline incorrect check in `util_tests`
cd2a4bc510 test: Redeclare variable as signed in `util_tests`
6f1c56f03a Merge bitcoin/bitcoin#35670: net: optimize compact block extra tx iteration
e94fda8a40 Merge bitcoin/bitcoin#35685: doc: Archive 30.3 release notes
5223cf1795 Merge bitcoin/bitcoin#35616: refactor: Use u64 over size_t for all cache sizes to fix a 32-bit overflow
c0e91efdb3 Merge bitcoin/bitcoin#35295: validation: fetch block input prevouts in parallel during ConnectBlock
f0da26cfc8 Merge bitcoin/bitcoin#34997: p2p: Don't participate in addr relay with feelers
443179a9eb doc: Archive 30.3 release notes
fabafd91f1 refactor: Use u64 over size_t for all cache sizes to fix a 32-bit overflow
32941e1314 Merge bitcoin/bitcoin#35649: depends: move FreeBSD SDK handling to CI
f379d716b1 Merge bitcoin/bitcoin#35669: doc: archive release notes for v31.1
881bb7b355 Merge bitcoin/bitcoin#35673: refactor: Move LoadGenesisBlock to ChainstateManager
3f3e644beb Merge bitcoin/bitcoin#35678: private broadcast: define and use new RPC_LIMIT_EXCEEDED error code ( + other follow-ups)
e3b026bf56 Merge bitcoin/bitcoin#34020: mining: add getTransactions(ByWitnessID) IPC methods
62f9089343 Merge bitcoin/bitcoin#35386: doc: add an AI contribution policy
fa615bd163 refactor: Move LoadGenesisBlock to ChainstateManager
8ac222484c private broadcast: remove no-op [[nodiscard]]
191bdcba26 test: align test better with described scenario
7ad311be18 test: use BOOST_CHECK_EQUAL for PrivateBroadcast::AddResult
82a02a2a22 rpc: define and use new  RPC_LIMIT_EXCEEDED error code
a64df338e6 Merge bitcoin/bitcoin#35651: doc: Improve offline-signing-tutorial after 32489
4498fa5d5b Merge bitcoin/bitcoin#35406: private broadcast: limit outstanding txs to count of 10,000
ec5edcd72b Merge bitcoin/bitcoin#35661: Update libmultiprocess subtree to add `ThreadMap.makePool` method
ea67fea062 Merge bitcoin/bitcoin#35464: kernel: Add function for creating chainparams with a signet challenge
a7b5f23bb8 Merge bitcoin/bitcoin#35667: refactor: Use `NetworkErrorString` for macOS code in `netif.cpp`
69fc991791 Merge bitcoin/bitcoin#32606: p2p: Drop unsolicited CMPCTBLOCK from non-HB peer and when blocksonly
1a3cbf1bd2 net: optimize compact block extra tx iteration
bc33509ae2 Merge bitcoin/bitcoin#35650: doc: Add release notes for 32489 (exportwatchonlywallet RPC)
2b6e767d96 doc: archive release notes for v31.1
b0735336ee p2p: Don't participate in addr relay with feeler connections
302733fd96 Merge bitcoin/bitcoin#35652: init: fix reindex deadlock by waking cv after interrupt
c1313b199f init: wake genesis wait after ImportBlocks() returns
eccb04a321 refactor: Use `NetworkErrorString` for macOS code in `netif.cpp`
f79ecfd9c7 Merge bitcoin/bitcoin#35658: refactor: Drop unneeded `<sys/types.h>` include before `<ifaddrs.h>`
22ac4ad949 ci: ensure we use correct lld version in OpenBSD job
495f43f7b3 ci: FreeBSD 15.1
244739db9d depends: move FreeBSD SDK handling to CI
bab0120053 Merge bitcoin/bitcoin#35621: validation: Ignore eventual error message from flushing in AcceptBlock
02afa66169 Merge commit '6b0a907302364649dcaffeb0340b985f14141b4e' into pr/subtree-11
6b0a907302 Squashed 'src/ipc/libmultiprocess/' changes from 3edbe8f67c1..16bf05dea02
22c328d388 refactor: Drop unneeded `<sys/types.h>` include before `<ifaddrs.h>`
2063f02bd5 Merge bitcoin/bitcoin#35510: test: SOCKS5 proxy: expect that connection may be reset during SOCKS5 handshake or data forwarding
1360001f43 Merge bitcoin/bitcoin#34959: wallet: Enforce BDB btree levels and overflow item sizes
10ffef4b3b Merge bitcoin/bitcoin#35604: log: expose `-logratelimit` in normal help
b393985aa0 Merge bitcoin/bitcoin#35634: txospenderindex: use zero-byte entry values
1835f2fcbf Merge bitcoin/bitcoin#35653: fuzz: Remove `ConsumeUniValue`
9f3e427228 fuzz: Remove ConsumeUniValue
68cb7840d2 doc: improve offline-signing-tutorial after 32489
32ddfc92d9 Merge bitcoin/bitcoin#35599: doc: Add release notes for #33671 (getbalances nonmempool field)
f56804bcf5 Merge bitcoin/bitcoin#35640: ci: use a 8x instance over 16x for riscv job
239d6c5260 Merge bitcoin/bitcoin#35614: HTTPServer: Prevent race condition between worker thread and I/O thread
cddbad325d doc: Add release notes for 32489 (exportwatchonlywallet RPC)
0a1bbec688 Merge bitcoin/bitcoin#32489: wallet: Add `exportwatchonlywallet` RPC to export a watchonly version of a wallet
4a007126fb Merge bitcoin/bitcoin#35147: depends: Boost 1.91.0-1
31abaa264c doc: add an AI contribution policy
a15bdc0598 doc: update offline-signing-tutorial to use exportwatchonlywallet rpc
a388076401 test: Test for exportwatchonlywallet
d053e3e5c8 wallet, rpc: Add exportwatchonlywallet RPC
444878efef wallet: Add CWallet::ExportWatchOnly
f9273f01db wallet: Move listdescriptors retrieving from RPC to CWallet
a1c83789a7 wallet: Write new descriptor's cache in AddWalletDescriptor
1e996640e6 wallet: Use Descriptor::CanSelfExpand() in CanGetAddresses()
d2ee9227da descriptor: Add CanSelfExpand()
2990bd7735 Merge bitcoin/bitcoin#35118: fuzz: add ipc round-trip fuzz target
fb1d152c24 depends: Boost 1.91.0-1
ba48852f9e Merge bitcoin/bitcoin#35438: test: introduce NodeSigner, run feature_taproot.py without wallet compiled
47bbed052e ci: use true|false over "true|false"
9a25bc3989 ci: use a 8x instance over 16x
9871dc7ab4 Merge bitcoin/bitcoin#31425: CI: Add Riscv bare metal job
113402286e doc: add txospenderindex release note
ce06878288 index: shrink txospenderindex value markers
a99148d576 test kernel: Don't log on warnings change
5b4fd284f4 kernel: Generate a signet with a challenge
037ad77071 fuzz: add IPC round-trip target
a8823c0996 Merge bitcoin/bitcoin#35607: nanobench: fix performance counter buffer initialization
f26c15bdd2 Merge bitcoin/bitcoin#35397: ci: add OpenBSD Clang cross job
91586f701e test: introduce NodeSigner, run feature_taproot.py without wallet compiled
771200ca43 test: return full keypair from `getnewdestination` helper
4e29de719e private broadcast: add release note for limited cap
cbf8c107c1 Release cs_main between individual private tx re-attempts
5aea3d0373 private broadcast: limit outstanding txs to count of 10,000
dc282ff31d Merge bitcoin/bitcoin#35597: logging: More fully remove libevent log category
ec98037f7a Merge bitcoin/bitcoin#35615: fuzz: restore CreateSock in PCP targets
43d89bb6e1 Merge bitcoin/bitcoin#35610: bitcoin-util: Add netmagic command
e35c805489 Merge bitcoin/bitcoin#35129: test: add fuzz test for private broadcast
dc1c17c085 doc: add release notes
0e10937184 fuzz: add coins_view_stacked fuzz harness to test concurrent leveldb reads
ce610a6ff4 fuzz: update harnesses to cover CoinsViewOverlay::StartFetching
760fb22dc3 test: add unit tests for CoinsViewOverlay::StartFetching
d69a3b20de doc: update CoinsViewOverlay docstring to describe parallel fetching
ab2a379237 coins: fetch inputs in parallel
fdf283036a coins: add ready flag to InputToFetch
ede11b8314 validation: collect block inputs in CoinsViewOverlay before ConnectBlock
f82043af50 coins: introduce thread pool in CoinsViewOverlay
5bf1c32008 validation: add -prevoutfetchthreads configuration option
a318f43254 bitcoin-util: Add netmagic command
256482ab56 validation: In AcceptBlock, ignore flush result
3c76bd4356 Merge bitcoin/bitcoin#35603: build: QRencode cleanups
f595daf1dd test: ensure HTTPServer race condition is fixed
b98b10c072 test: introduce a worker thread in http socket error test
922b08d375 test: socket error handling in HTTPServer using ErrorSock mock socket
73da2a8a52 http: prevent race condition between worker thread and I/O thread
57b3bf8496 Merge bitcoin/bitcoin#35609: ci: Bump tsan config to ubuntu:26.04 with -U_FORTIFY_SOURCE
d64ea15824 ci: add openBSD cross CI job
5404b62074 depends: add openbsd_LDFLAGS
2a9e35d293 Merge bitcoin/bitcoin#35588: scripted-diff: Rename `Sock::{RECV,SEND,ERR}`
a0e5e30010 fuzz: restore CreateSock in PCP targets
41ceea400e scripted-diff: Rename `StatusLevel::{INFO,WARN,ERR}`
f395acdeee scripted-diff: Rename `Sock::{RECV,SEND,ERR}`
7a74f65293 Merge bitcoin/bitcoin#35536: fuzz: share a single mocked steady clock across FuzzedSock instances
e1290ce7f7 Merge bitcoin/bitcoin#35543: test: introduce ExtendedPrivateKey and ExtendedPublicKey classes
7ac25c9177 util, refactor: Rename local `ERR` in `Sock::Accept`
bbbbab86a8 ci: Bump tsan config to ubuntu:26.04 with -U_FORTIFY_SOURCE
095596ddf7 log: expose -logratelimit option
ea9afb61a1 Merge bitcoin/bitcoin#35602: doc: Clarify build docs about `pkgconf` / `pkg-config` requirements
672eedc46b Merge bitcoin/bitcoin#35220: fuzz: connman: strengthen assertions and extend coverage
0e5c718d8a Merge bitcoin/bitcoin#35506: test: ensure group data cluster pointers are live
b6b1d0653a nanobench: fix perf counter buffer init
93012d7ff9 Merge bitcoin/bitcoin#35601: wallet: remove experimental warning from send and sendall
829255c8be cmake: Remove `SelectLibraryConfigurations` from `FindQRencode` module
5c55606da9 depends: Remove unused `lib/pkgconfig` in `qrencode` package
402ba10b20 cmake: Drop optional `PkgConfig` use in `FindQRencode` module
8ebfff0f88 doc: add send RPC release note
fb8a103886 doc: Clarify build docs about `pkgconf` / `pkg-config` requirements
5884f5a4fa wallet: remove experimental warning from send RPCs
7b84e5106c Merge bitcoin/bitcoin#35595: ci: remove some packages from Chimera job
58560c281d ci: remove some packages from Chimera job
295ce6f45c Merge bitcoin/bitcoin#35576: test: raise `feature_reindex` RPC timeout
b36730a3ef  Add CI job for riscv bare metal
bfdbf513f6 Add CI job for producing a static bare metal binary
a9a1d92a1d build: Add option for building for bare metal envs
9b2ef81757 doc: add release notes for #33671 (getbalances nonmempool field)
633044f143 Merge bitcoin/bitcoin#35266: rpc, wallet: add an option to not load the wallet after migrating
d6269e2a90 Merge bitcoin/bitcoin#35594: fuzz: cover async chainstate compaction
3765b428d1 logging: More fully remove libevent log category
703a671fbc fuzz: compact coins view db during fuzzing
0868c85fd5 refactor: rename async coin compaction
6fa4132298 fuzz: share a single mocked steady clock across FuzzedSock instances
8791c4764c test: use ExtendedPrivateKey in wallet_taproot.py
89ceafafb9 test: use ExtendedPrivateKey in wallet_listdescriptors.py
bbfffcab58 test: use ExtendedPrivateKey in wallet_send.py
2ab6e590f7 test: use ExtendedPrivateKey in wallet_keypool.py
9e20118720 test: use ExtendedPrivateKey in wallet_fundrawtransaction.py
06af0cddbb test: use ExtendedPrivateKey in wallet_descriptor.py
4100fac20e test: use ExtendedPrivateKey in wallet_createwallet.py
ff3f6def9a test: use ExtendedPrivateKey in wallet_bumpfee.py
003f2a01f6 test: use ExtendedPrivateKey in feature_notifications.py
f988e6d6e6 test: use ExtendedPrivateKey in wallet_importdescriptors.py
0cdd817a82 add release note
517d37ce3e test: tests wallet migration with load_wallet disabled
b98dd63da7 rpc: Add load_wallet argument to migratewallet RPC
4acd063ba6 wallet: make loading the wallet after migrating optional
1a3cfdf1b7 fuzz: connman: cover AddLocalServices/RemoveLocalServices
c507fb3063 fuzz: connman: add outbound-bytes invariants
4a6fce43ea fuzz: connman: add AddNode/RemoveAddedNode invariants
9e6546c517 test: raise reindex mining RPC timeout
d2a03d50ac test: add extendedkey.py unit tests by using BIP32 test vectors
afdb378082 test: introduce ExtendedPrivateKey and ExtendedPublicKey classes
55e3a57f22 qa: Avoid UTXO reuse between test functions
9c5dd2926a p2p: Ignore CMPCTBLOCK from peer that hasn't sent SENDCMPCT
bf9884f4e5 p2p: make blocksonly nodes ignore CMPCTBLOCK messages
92cea63c71 test: (Un)solicited invalid cb -> get disconnected.
e845e26344 test: p2p: Nodes ignore unsolicited CMPCTBLOCK's
8313591715 p2p: Drop unsolicited CMPCTBLOCK from non-HB peer
44f377a71f refactor: test: Static assert_highbandwidth_states
25457a3272 test: Tighten getblocktxn checks in parallel cb reconstruction test.
2ee4fafa3f test: add fuzz test for private broadcast
08b7c61fc7 private broadcast: enforce sending to unique node ids
9a8ef9b0a3 test: SOCKS5 proxy: expect that connection may be reset during handshake
eb3208364a test: SOCKS5 proxy: expect that connection may be reset when forwarding
4dbaa7cc65 test: generalise byte_to_base58 utility function to allow more version types
df9eb72b12 test: ensure group data cluster pointers are live
a5859edef4 fuzz: connman: set m_local_services/m_use_addrman_outgoing/m_max_automatic_connections
4b84c9125a fuzz: connman: add network activity invariants
9784818442 mining: add getTransactionsByWitnessID() IPC method
d282ae6883 mining: add getTransactionsByTxID() IPC method
0d5e4d4712 test: restart node after IPC option override test
f16b3613cd ipc: Serialize null CTransactionRef as empty Data
0f466e1094 mempool: add lookup by witness hash
51dd90fb50 refactor: Merge announce_cmpct_block() defs into one
97d08d62ba refactor: store wallet names to MigrationResult
8a739a5510 build: allow ipc fuzz builds
b2de59d486 wallet, bdbro: Validate btree page levels
dc3a2b9c3b wallet, bdbro: Enforce overflow data lengths

git-subtree-dir: libbitcoinkernel-sys/bitcoin
git-subtree-split: 81405fc7abbd1889f3978b8924e7acbe12b3403b
Kino1994 pushed a commit to Kino1994/bitcoin-full-history that referenced this pull request Jul 10, 2026
6c21d69 doc: add txospenderindex release note (Lőrinc)
1ced471 index: shrink txospenderindex value markers (Lőrinc)

Pull request description:

  **Problem:** `TxoSpenderIndex` values are never read: `FindSpender()` only uses the key, which already encodes the spender's disk position, but the values are written as single-byte `\0`(instead of empty values).

  **Fix:** Write zero-byte markers for new `txospenderindex` entries instead.
  Existing indexes stay readable because both formats use the same keys and the value is ignored.
  Rebuilding is only needed to shrink old entries.

  **Related work:** This complements #35568, which saves another ~4.5 GB by dropping bloom filters.
  A #35531-style rewrite could let existing indexes reclaim both savings without a full rebuild.

  **Reproducer:** Automated tests weren't added, but the manual reproducer below builds both commits, checks old/new format compatibility through `gettxspendingprevout`, and prints rebuilt index sizes.
  On my mainnet datadir this PR shrank `txospenderindex` by ~3.4 GB (from 89G to 86G).

  <details><summary>Script + sample output</summary>

  ```bash
  BEFORE="89b4000ae06c72c5a14ee05ad70d3aece3f1b382" AFTER="1be799ef858c33158a302857b084e169b5bf8c1f" DATA_DIR="/mnt/my_storage/BitcoinData" LOG="${DATA_DIR}/debug.log" OUT='[{"txid":"0437cd7f8525ceed2324359c2d0ba26006d92d856a9c20fa0241106ee5a597c9","vout":0}]' SPEND="f4184fc596403b9d638783cf57adfe4c75c605f6356fbc91338530e9831e9e16"; \
  for b in before after; do [ -x ./build-$b/bin/bitcoin-cli ] && ./build-$b/bin/bitcoin-cli -datadir="${DATA_DIR}" stop >/dev/null 2>&1 || true; done; sleep 10; \
  git reset --hard >/dev/null 2>&1 && git clean -fxd >/dev/null 2>&1 && (git fetch origin "$BEFORE" "$AFTER" >/dev/null 2>&1 || true) && \
  for c in before:$BEFORE after:$AFTER; do git checkout ${c#*:} >/dev/null 2>&1 && cmake -B build-${c%:*} -DCMAKE_BUILD_TYPE=Release >/dev/null 2>&1 && cmake --build build-${c%:*} -j --target bitcoind bitcoin-cli >/dev/null 2>&1; done && \
  for b in before after; do ./build-$b/bin/bitcoin-cli -datadir="${DATA_DIR}" stop >/dev/null 2>&1 || true; done; sleep 10; \
  start_node() { [ "$2" = wipe ] && rm -rf "${DATA_DIR}/indexes/txospenderindex" "${LOG}"; : > "${LOG}"; ./build-"$1"/bin/bitcoind -datadir="${DATA_DIR}" -txospenderindex=1 -connect=0 -printtoconsole=0 & pid=$!; while ! grep -Fq 'txospenderindex is enabled at height' "${LOG}" 2>/dev/null; do kill -0 "$pid" 2>/dev/null || { tail -100 "${LOG}"; return 1; }; sleep 5; done; }; \
  check_spend() { result="$(./build-"$1"/bin/bitcoin-cli -datadir="${DATA_DIR}" gettxspendingprevout "$OUT" '{"mempool_only":false}')"; echo "$2: $result"; echo "$result" | grep -q "$SPEND"; }; \
  stop_node() { ./build-"$1"/bin/bitcoin-cli -datadir="${DATA_DIR}" stop >/dev/null; wait "$pid"; }; \
  echo "prevout: 0437cd7f8525ceed2324359c2d0ba26006d92d856a9c20fa0241106ee5a597c9:0 -> ${SPEND}" && \
  start_node before wipe && check_spend before "old binary reads old-format index" && du -sh "${DATA_DIR}/indexes/txospenderindex"; stop_node before && \
  start_node after && check_spend after "new binary reads old-format index"; stop_node after && \
  start_node after wipe && check_spend after "new binary reads new-format index" && du -sh "${DATA_DIR}/indexes/txospenderindex"; stop_node after && \
  start_node before && check_spend before "old binary reads new-format index"; stop_node before
  ```

  Expected result, trimmed to the relevant compatibility and size lines:

  ```text
  prevout: 0437cd7f8525ceed2324359c2d0ba26006d92d856a9c20fa0241106ee5a597c9:0 -> f4184fc596403b9d638783cf57adfe4c75c605f6356fbc91338530e9831e9e16
  old binary reads old-format index: [ ... "spendingtxid": "f4184fc596403b9d638783cf57adfe4c75c605f6356fbc91338530e9831e9e16" ... ]
  89G     /mnt/my_storage/BitcoinData/indexes/txospenderindex
  new binary reads old-format index: [ ... "spendingtxid": "f4184fc596403b9d638783cf57adfe4c75c605f6356fbc91338530e9831e9e16" ... ]
  new binary reads new-format index: [ ... "spendingtxid": "f4184fc596403b9d638783cf57adfe4c75c605f6356fbc91338530e9831e9e16" ... ]
  86G     /mnt/my_storage/BitcoinData/indexes/txospenderindex
  old binary reads new-format index: [ ... "spendingtxid": "f4184fc596403b9d638783cf57adfe4c75c605f6356fbc91338530e9831e9e16" ... ]
  ```
  </details>

ACKs for top commit:
  davidgumberg:
    crACK bitcoin/bitcoin@6c21d69
  sedited:
    ACK 6c21d69
  ekzyis:
    utACK 6c21d69
  andrewtoth:
    ACK 6c21d69

Tree-SHA512: 07945b9470670115e27a1ef51e64164da7b91e550fdcb59952bfc98d862264b405e8fd8c3267c42718a39823f853057390d031d936d6e0b2172ec07e36cfdd67
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants