wallet: Use in-memory SQLite for temporary wallet in exportwatchonlywallet#35655
Conversation
|
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers. Code Coverage & BenchmarksFor details see: https://corecheck.dev/bitcoin/bitcoin/pulls/35655. ReviewsSee the guideline and AI policy for information on the review process.
If your review is incorrectly listed, please copy-paste ConflictsReviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first. LLM Linter (✨ experimental)Possible places where comparison-specific test macros should replace generic comparisons:
2026-07-08 22:04:25 |
ebfee85 to
532c677
Compare
|
Concept ACK Code review 532c677. My LLM friend found a potential issue: when Further more, consider the following patch:
diff --git a/src/wallet/sqlite.h b/src/wallet/sqlite.h
index 33c9628c49..4709967d39 100644
--- a/src/wallet/sqlite.h
+++ b/src/wallet/sqlite.h
@@ -178,5 +178,5 @@ class InMemoryWalletDatabase : public SQLiteDatabase
public:
InMemoryWalletDatabase();
- std::string Filename() override { return "memory"; }
+ std::string Filename() override { return "<in-memory>"; }
std::vector<fs::path> Files() override { return {}; }
};
diff --git a/src/wallet/wallet.cpp b/src/wallet/wallet.cpp
index be618efbd6..6714b4ad95 100644
--- a/src/wallet/wallet.cpp
+++ b/src/wallet/wallet.cpp
@@ -3882,8 +3882,8 @@ bool CWallet::MigrateToSQLite(bilingual_str& error)
}
- // Close this database and delete the file
- fs::path db_path = fs::PathFromString(m_database->Filename());
+ // Close this database and delete its file(s) - just one for BerkeleyRO
+ const std::vector<fs::path> old_files = m_database->Files();
m_database->Close();
- fs::remove(db_path);
+ for (const fs::path& file : old_files) fs::remove(file);
// Generate the path for the location of the migrated wallet
@@ -3909,5 +3909,5 @@ bool CWallet::MigrateToSQLite(bilingual_str& error)
batch->TxnAbort();
m_database->Close();
- fs::remove(m_database->Filename());
+ for (const fs::path& file : m_database->Files()) fs::remove(file);
assert(false); // This is a critical error, the new db could not be written to. The original db exists as a backup, but we should not continue execution.
}
@@ -4418,5 +4418,7 @@ util::Result<MigrationResult> MigrateLegacyToDescriptor(std::shared_ptr<CWallet>
// Wallets stored directly as files in the top-level directory
// (e.g. default unnamed wallets) don’t have a removable parent directory.
- wallet_empty_dirs_to_remove.insert(fs::PathFromString(wallet.GetDatabase().Filename()).parent_path());
+ if (!files.empty()) {
+ wallet_empty_dirs_to_remove.insert(files.front().parent_path());
+ }
}
}; |
I'll add a fix for that on a separate commit (first/ base) as suggested, thanks!
I agree, will change it.
I agree with this also, but I'll do on a separate PR because I found that there are even more places to change (everything is within src/wallet/):
On the migration fixes (
Although they're independent of this PR, I think these are worth fixing and I'll add a 3rd commit with it. Thank you! |
532c677 to
0c276ad
Compare
…abase `WalletDatabase::Filename()` is documented as returning a path "for logs and error messages", but several callers use it for file operations (removal, path derivation), which is incorrect — they should use `Files()` instead. Renaming to `DisplayFileName()` makes the display-only intent explicit at every call site, making it immediately clear when a caller is misusing it for file operations. This is a pure rename: 1 pure virtual declaration, 3 overrides (`SQLiteDatabase`, `MigrateDatabase`, `MockableSQLiteDatabase`), and all callers. Callers that currently misuse the result for file operations are left in place; fixing those by switching to `Files()` is a separate concern being addressed in bitcoin#35655. No behavior change. Suggested by Sjors during review of bitcoin#35655. Co-Authored-By: Sjors Provoost <sjors@sprovoost.nl>
|
-Updates:
|
There was a problem hiding this comment.
Code review b67d6b5
I think MockableSQLiteDatabase should inherit from InMemoryWalletDatabase. That should also let you introduce the new subclass in its own commit, before _wallet: use in-memory SQLite for temporary wallet in exportwatchonlywallet_.
In 7b6136e wallet: store m_additional_flags in SQLiteDatabase to fix reopen path: there's a second problem that we should fix: it makes no sense to Close() and Open() an in-memory database, that's just going to wipe everything. It could throw() instead.
In b67d6b5 wallet, refactor: rename Filename() to DisplayFileName() in WalletDatabase: I think we should go one step further and set m_display_file_name at construction time. That's safer because the SQLiteDatabase constructor logs things, so we shouldn't rely on an override DisplayFileName(). It also ties into the next suggestions.
Note that the Files() helper, introduced as part of #31423, is just guessing which files exist. You could make that a bit more robust by using sqlite3_filename_journal in modern sqlite3 versions.
We also rely on InMemoryWalletDatabase to override Files() with an empty list. Instead we could make m_file_path and m_journal_file_path optional, and not set them for an in-memory wallet. I have a commit that does this, along with some constructor ergonomics.
Here's a branch that does all the above (except for splitting b67d6b5).
There was a problem hiding this comment.
Concept ACK b67d6b5
PR Uses the functionality of SQLite to create a in-memory database to use as the temporary storage func. of the WatchOnlyWallet. This has the advantage that it's really temporary and removes a lot of removal logic from the code. Which I think is an improvement.
NIT: The changed open function with stored additional flags can use a regression test (imho)
eg:
Details
something like this, as additional test in db_tests.cpp:
BOOST_AUTO_TEST_CASE(in_memory_connection_refresh)
{
// Regression test for the reopen path of an in-memory database.
InMemoryWalletDatabase database;
std::string key = "key";
std::string value = "value";
// Reopening an in-memory database must not attempt any on-disk access.
database.Close();
BOOST_CHECK_NO_THROW(database.Open());
// The refreshed in-memory connection must remain usable for reads and writes.
SQLiteBatch batch{database};
BOOST_CHECK(batch.Write(key, value));
std::string read_value;
BOOST_CHECK(batch.Read(key, read_value));
BOOST_CHECK_EQUAL(read_value, value);
}EDIT: i Agree with Sjors that throwing would be a better solution that recovering to a empty state.
achow101
left a comment
There was a problem hiding this comment.
Filename()is documented as returning a path for display/logging only, but the file-op callers above made its misuse invisible. After switching those toFiles(),Filename()is renamed toDisplayFileName()to enforce the display-only intent at every remaining call site.
I think this is a mischaracterization of Filename(). When the documentation was originally written, it's only use was for logging, where we want to log the full path to the data file. Just because the documentation still says that does not mean that it is exclusively to be used for logging, nor that using it outside of logging is misuse. Filename() must point to the primary data file and callers can assume that it will be a full path to that file. I don't think this should be changed at all.
b67d6b5 "wallet, refactor: rename Filename() to DisplayFileName() in WalletDatabase" Should be dropped. I'm inclined to say that 0c276ad "wallet: use Files() instead of Filename() for file removal in migration" should be dropped as well. Both of these are also mostly unrelated to the main change in this PR.
No, every caller needs to intentionally choose what to put for the paths and names as these are relevant to sqlite's usage. We should not be defaulting names or paths as they may result in the wrong database files being opened. |
b67d6b5 to
176797a
Compare
Done the first two — On |
176797a to
ce48d55
Compare
Thanks for the ACK and the regression test suggestion! Added it in a separate commit, co-authored by you. Since the behaviour changed from silently reopening to throwing, the assertion is inverted: |
Agreed, keeping the explicit |
|
-Updates:
|
|
@achow101 wrote:
Then let's actually document the function to say so. But it's inconsistent with then setting it to
What do you mean by this? |
You're literally allowing file path to be nullopt, and then choosing an default when it is, and passing that default in to If a subclass wants to use a bogus file path, it must do so explicitly by choice. |
08ca2a5 to
3165ed8
Compare
|
-Updates:
|
|
utACK 3165ed8 @achow101 thanks for the clarification.
The default should have been the magic string
The commit enforces The fact that Latest version of what I had in mind: Sjors@7c6026c That said, I think the |
Thanks for working on that idea, I find it interesting and I like the approach. I noticed some details (e.g. magic string I think should live and belongs to |
SQLiteDatabase::Open() (the public override) always reopens the database with no additional flags. If SQLiteBatch::Close() triggers the force_conn_refresh path (TxnAbort failed), it calls Open() which drops the original additional_flags, causing in-memory databases to be reopened as on-disk instead. Store additional_flags as a member and use it in Open() so the reconnect preserves the original flags. For in-memory databases, connection recovery makes no sense as all data would be lost; both the force_conn_refresh path and the public Open() now throw instead. Co-authored-by: Sjors Provoost <sjors@sprovoost.nl>
3165ed8 to
66d4d97
Compare
…allet
The intermediate watchonly wallet created during exportwatchonlywallet is
a pure build artifact — it is always discarded once BackupWallet() copies
it to the destination. Creating it as an in-memory SQLiteDatabase
(SQLITE_OPEN_MEMORY) removes the need to write files to the wallets
directory and eliminates the cleanup handler that deleted those files on
both success and failure paths.
Introduces InMemoryWalletDatabase (a minimal SQLiteDatabase subclass) and
MakeInMemoryWalletDatabase() factory in sqlite.h/cpp, following the same
pattern as MockableSQLiteDatabase / CreateMockableWalletDatabase() in the
test utilities. MockableSQLiteDatabase now derives from InMemoryWalletDatabase,
removing its redundant Files() override.
The wallet is named after the source wallet ("<name>_watchonly_temp") so
concurrent exports of different wallets use distinct names and log lines
remain traceable to the source wallet.
InMemoryWalletDatabase::Open() now throws to prevent silently returning a fresh empty connection after close, which would discard all data. Add a test to pin this behaviour. Co-authored-by: Jan B <608446+janb84@users.noreply.github.com>
66d4d97 to
777d23f
Compare
|
re-utACK 777d23f Just a text change since my previous review.
Yes, that's fine. |
|
ACK 777d23f |
…ry wallet in exportwatchonlywallet 5a96701 test: add regression test for in-memory SQLiteDatabase reopen (Pablo Martin) 7b90483 wallet: use in-memory SQLite for temporary wallet in exportwatchonlywallet (Pablo Martin) 194403b wallet: store m_additional_flags in SQLiteDatabase to fix reopen path (Pablo Martin) Pull request description: Since #33032 landed (in-memory `SQLiteDatabase` via `SQLITE_OPEN_MEMORY`), the intermediate wallet built during `exportwatchonlywallet` can live entirely in memory instead of being written to the wallets directory as a temporary file. The temp wallet is a pure build artifact: it is populated with descriptors, transactions, and address book data, then immediately discarded once `BackupWallet()` copies its contents to the destination file. Making it in-memory removes all on-disk footprint and eliminates the `cleanup_watchonly_wallet` RAII handler — along with the `wallet_path` and `cleanup_files` variables it needed — which previously ensured the temp files were deleted on both success and failure paths. This PR introduces `InMemoryWalletDatabase` (a minimal `SQLiteDatabase` subclass) and `MakeInMemoryWalletDatabase()` factory in `sqlite.h/cpp`, following the same pattern as `MockableSQLiteDatabase` / `CreateMockableWalletDatabase()`. `MockableSQLiteDatabase` now derives from `InMemoryWalletDatabase`, removing its redundant `Files()` override. Suggested by Sjors in #32489 ([comment](bitcoin/bitcoin#32489 (comment))). --- Also fixes a related issue found (by Sjors) during review: - `SQLiteDatabase::Open()` (the no-arg public override) hardcoded 0 as `additional_flags` when reopening after a failed `TxnAbort()`, which would reopen an in-memory database as on-disk. Fixed by storing `m_additional_flags` in the constructor and using it in the reopen path. For in-memory databases, both the `force_conn_refresh` path and the public `Open()` now throw instead of silently creating a fresh empty connection. A regression test for the `Open()` throw is included in a separate commit. --- As a follow-up, `InMemoryWalletDatabase` could replace `MockableSQLiteDatabase` in `src/bench/` (5 files, 6 call sites), since benchmarks don't need mock-specific behaviour and benefit from using the same in-memory path as production code. ACKs for top commit: Sjors: re-utACK 5a96701 achow101: ACK 5a96701 janb84: ACK 5a96701 Tree-SHA512: 71178ce99c7ebc0fc5ba17956c27d37f90e3d36cefbdc0d15d1424b7a70bf15f05a13cb03c268d885678aba1fd3c90567d8389d3680650c8ae46f4cb4b10b28a
…tabase Benchmarks don't need mock-specific behaviour (overridden Filename(), Format(), or the exposed batch-level WriteKey()). Replace CreateMockableWalletDatabase() with MakeInMemoryWalletDatabase() across 6 call sites in src/bench/ (5 files), using the same in-memory SQLite path that production code uses. wallet_migration.cpp is excluded: it calls GetOrCreateLegacyDataSPKM() which asserts Format() == "sqlite-mock", a signal used to allow legacy SPKM setup in test/bench contexts. MockableSQLiteDatabase is still correct there. For coin_selection.cpp, which had no other dependencies on wallet/test/util.h, also switch the include to <wallet/sqlite.h>. Follow-up suggested in bitcoin#35655.
…tabase Benchmarks don't need mock-specific behaviour (overridden Filename(), Format(), or the exposed batch-level WriteKey()). Replace CreateMockableWalletDatabase() with MakeInMemoryWalletDatabase() across 6 call sites in src/bench/ (5 files), using the same in-memory SQLite path that production code uses. wallet_migration.cpp is excluded: it calls GetOrCreateLegacyDataSPKM() which asserts Format() == "sqlite-mock", a signal used to allow legacy SPKM setup in test/bench contexts. MockableSQLiteDatabase is still correct there. For coin_selection.cpp, which had no other dependencies on wallet/test/util.h, also switch the include to <wallet/sqlite.h>. Follow-up suggested in bitcoin#35655.
…MemoryWalletDatabase 7508ac3 bench: replace CreateMockableWalletDatabase with MakeInMemoryWalletDatabase (Pablo Martin) Pull request description: Benchmarks don't need mock-specific behaviour (overridden `Filename()`, `Format()`, or the exposed batch-level `WriteKey()`). Replace `CreateMockableWalletDatabase()` with `MakeInMemoryWalletDatabase()` across all 6 call sites in `src/bench/` (5 files - 4 wallet bench files + 1 in coin_selection), using the same in-memory SQLite path that production code uses. `wallet_migration.cpp` is excluded: it calls `GetOrCreateLegacyDataSPKM()` which asserts `Format() == "sqlite-mock"`, a deliberate signal that allows legacy SPKM setup in `test/bench` contexts. `MockableSQLiteDatabase` is still correct there. For `coin_selection.cpp`, which had no other dependencies on `wallet/test/util.h`, the include is switched to `<wallet/sqlite.h>`. The remaining 4 files retain `wallet/test/util.h` for other utilities but also add an explicit `<wallet/sqlite.h>` include as required by IWYU. Follow-up suggested in #35655. ACKs for top commit: sedited: ACK 7508ac3 janb84: ACK 7508ac3 Tree-SHA512: 6f4086eb5700ba3da882378dcee76b3c075670cdea5d71b45c9b369e48c72b8f591e0c5a6bd4504719dd512b3b7d2b23230a8b76e471c6c0571c13325fa3cba0
…4e48e968a 114e48e968a kernel: Add script tracer 54e1a95a12e Merge bitcoin/bitcoin#35719: ci: disable Qt build in OpenBSD cross job b0e09511586 ci: disable Qt build in OpenBSD cross job 734c34bafda Merge bitcoin/bitcoin#35427: depends: Build `qt` and `qrencode` packages on OpenBSD ee61b11a9e7 Merge bitcoin/bitcoin#35200: node: smooth oversized `dbcache` warnings e3554bf361f Merge bitcoin/bitcoin#35579: wallet: reserve walletrescan before checking wallet is at the tip 11ae4265522 Merge bitcoin/bitcoin#35715: cmake: Fix WITH_EXTERNAL_LIBMULTIPROCESS + BUILD_FUZZ_BINARY e544413c0de Merge bitcoin/bitcoin#32763: wallet: Replace CWalletTx::mapValue and vOrderForm with explicit class members fe1cb6e40d7 Merge bitcoin/bitcoin#35690: wallet: Introduce WalletError with machine-readable error code 441f3114f57 Merge bitcoin/bitcoin#35659: Clarify supported *BSD releases and drop outdated workarounds 0399df827c6 Merge bitcoin/bitcoin#35708: depends: capnp 1.5.0 1ab1fdd4696 Merge bitcoin/bitcoin#35705: bench: replace CreateMockableWalletDatabase with MakeInMemoryWalletDatabase db35b9238fc ipc # build: Fix fuzz target CMakeLists.txt for external libmultiprocess d18fec892e2 Merge bitcoin/bitcoin#35698: doc: Update enum class constant naming style guide a2e4cd7ad2a depends: capnp 1.5.0 7508ac319d9 bench: replace CreateMockableWalletDatabase with MakeInMemoryWalletDatabase 907e284e303 Merge bitcoin/bitcoin#35701: test: Remove `mock_process.cpp` c8459b6bdcd Merge bitcoin/bitcoin#35568: txospenderindex: disable bloom filters to optimize disk usage 63c5f9d22c0 test: Remove `mock_process.cpp` ef101b04a8d Merge bitcoin/bitcoin#35655: wallet: Use in-memory SQLite for temporary wallet in exportwatchonlywallet b6becf3534c Merge bitcoin/bitcoin#35684: Update libmultiprocess subtree to add `max_connections` option 930f25050f7 Merge bitcoin/bitcoin#35700: doc: archive release notes for v29.4 9b2b3f4ec6f doc: archive release notes for v29.4 297fd1489bb Merge bitcoin/bitcoin#35412: ci: add NetBSD Clang cross job e314869066b Merge bitcoin/bitcoin#35695: Remove myself as security contact e3d67a5eae5 Merge bitcoin/bitcoin#35691: chainparams: delete my DNS seed a8223bb4e62 wallet: Introduce WalletError with machine-readable error code fad5809cb92 doc: Update enum class constant naming style guide 7d8137c1417 Merge bitcoin/bitcoin#34897: indexes: Don't commit ahead of the flushed chainstate 629df81e4c6 Remove myself as security contact d164a043426 node: smooth oversized `-dbcache` warnings d9080639804 chainparams: delete my DNS seed e9ed898a0da validation: Don't use m_chain.Tip() in FlushStateToDisk 3679f1ecf5e index: Don't commit ahead of the flushed chainstate 65735728a5a index: Remove return value from Commit() 09c06960c6a validation: track last flushed block 13c02b5466d test: add test for index commits ahead of the last flushed block c43b7a1115f ci: add netBSD cross CI job 699c21aea47 depends: add netbsd_LDFLAGS 777d23f25c7 test: add regression test for in-memory SQLiteDatabase reopen d1e7f8c986f wallet: use in-memory SQLite for temporary wallet in exportwatchonlywallet ee43743f126 wallet: store m_additional_flags in SQLiteDatabase to fix reopen path a9d1b652f32 Merge commit '707d0ded84563386f770ec17970834c65f8fa938' into pr/subtree-12 707d0ded845 Squashed 'src/ipc/libmultiprocess/' changes from 16bf05dea02..28e056576a3 2bab6bc73f2 refactor: Drop support for FreeBSD < 14 91b5c8a07c6 refactor: Remove FreeBSD-specific workaround 56701ff6d5c doc: Clarify supported *BSD releases 6d0ea4cf5bd doc: add release notes a2b1c86903a txospenderindex: disable bloom filters to optimize disk usage fed3cf6f0ed wallet: Replace CWalletTx's vOrderForm with specific fields 4f8823e8e11 wallet: Drop vOrderForm from CommitTransaction 9e62e4b1f34 test: slow down rescaning process a2b0bfcd854 wallet: Drop mapValue from CWalletTx cb99864c91c wallet: Throw if unknown entry is found in mapValue 98d5cdae663 wallet: Make CWalletTx "replaces_txid" and "replaced_by_txid" member variables 7ef8a6efc2a wallet: Make CWalletTx "comment" and "to" member variables 2155e913d3e wallet: Make CWalletTx "from" and "message" member variables c6ba98dcc8a wallet: Drop mapValue from CommitTransaction 00abb174a80 wallet: Pass comment and comment_to to CommitTransaction 1a219a37a21 wallet: Pass replaces_txid to CommitTransaction outside of mapValue 336f5a738b3 wallet: reserve walletrescan before checking wallet is at the tip a54ec373a69 depends: Build `qt` and `qrencode` packages for OpenBSD hosts git-subtree-dir: libbitcoinkernel-sys/bitcoin git-subtree-split: 114e48e968a087f74c1ab611ac2a31a9266812e1
Since #33032 landed (in-memory
SQLiteDatabaseviaSQLITE_OPEN_MEMORY), the intermediate wallet built duringexportwatchonlywalletcan live entirely in memory instead of being written to the wallets directory as a temporary file.The temp wallet is a pure build artifact: it is populated with descriptors, transactions, and address book data, then immediately discarded once
BackupWallet()copies its contents to the destination file. Making it in-memory removes all on-disk footprint and eliminates thecleanup_watchonly_walletRAII handler — along with thewallet_pathandcleanup_filesvariables it needed — which previously ensured the temp files were deleted on both success and failure paths.This PR introduces
InMemoryWalletDatabase(a minimalSQLiteDatabasesubclass) andMakeInMemoryWalletDatabase()factory insqlite.h/cpp, following the same pattern asMockableSQLiteDatabase/CreateMockableWalletDatabase().MockableSQLiteDatabasenow derives fromInMemoryWalletDatabase, removing its redundantFiles()override.Suggested by Sjors in #32489 (comment).
Also fixes a related issue found (by Sjors) during review:
SQLiteDatabase::Open()(the no-arg public override) hardcoded 0 asadditional_flagswhen reopening after a failedTxnAbort(), which would reopen an in-memory database as on-disk. Fixed by storingm_additional_flagsin the constructor and using it in the reopen path. For in-memory databases, both theforce_conn_refreshpath and the publicOpen()now throw instead of silently creating a fresh empty connection. A regression test for theOpen()throw is included in a separate commit.As a follow-up,
InMemoryWalletDatabasecould replaceMockableSQLiteDatabaseinsrc/bench/(5 files, 6 call sites), since benchmarks don't need mock-specific behaviour and benefit from using the same in-memory path as production code.