Skip to content

wallet: Use in-memory SQLite for temporary wallet in exportwatchonlywallet#35655

Merged
achow101 merged 3 commits into
bitcoin:masterfrom
pablomartin4btc:wallet/exportwatchonly-inmemory-tmp-wallet
Jul 10, 2026
Merged

wallet: Use in-memory SQLite for temporary wallet in exportwatchonlywallet#35655
achow101 merged 3 commits into
bitcoin:masterfrom
pablomartin4btc:wallet/exportwatchonly-inmemory-tmp-wallet

Conversation

@pablomartin4btc

@pablomartin4btc pablomartin4btc commented Jul 4, 2026

Copy link
Copy Markdown
Member

Since #33032 landed (in-memory SQLiteDatabase via SQLITE_OPEN_MEMORY), the intermediate wallet built during exportwatchonlywallet can live entirely in memory instead of being written to the wallets directory as a temporary file.

The temp wallet is a pure build artifact: it is populated with descriptors, transactions, and address book data, then immediately discarded once BackupWallet() copies its contents to the destination file. Making it in-memory removes all on-disk footprint and eliminates the cleanup_watchonly_wallet RAII handler — along with the wallet_path and cleanup_files variables it needed — which previously ensured the temp files were deleted on both success and failure paths.

This PR introduces InMemoryWalletDatabase (a minimal SQLiteDatabase subclass) and MakeInMemoryWalletDatabase() factory in sqlite.h/cpp, following the same pattern as MockableSQLiteDatabase / CreateMockableWalletDatabase(). MockableSQLiteDatabase now derives from InMemoryWalletDatabase, removing its redundant Files() override.

Suggested by Sjors in #32489 (comment).


Also fixes a related issue found (by Sjors) during review:

  • SQLiteDatabase::Open() (the no-arg public override) hardcoded 0 as additional_flags when reopening after a failed TxnAbort(), which would reopen an in-memory database as on-disk. Fixed by storing m_additional_flags in the constructor and using it in the reopen path. For in-memory databases, both the force_conn_refresh path and the public Open() now throw instead of silently creating a fresh empty connection. A regression test for the Open() throw is included in a separate commit.

As a follow-up, InMemoryWalletDatabase could replace MockableSQLiteDatabase in src/bench/ (5 files, 6 call sites), since benchmarks don't need mock-specific behaviour and benefit from using the same in-memory path as production code.

@DrahtBot DrahtBot added the Wallet label Jul 4, 2026
@DrahtBot

DrahtBot commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/35655.

Reviews

See the guideline and AI policy for information on the review process.

Type Reviewers
ACK Sjors, achow101, janb84

If your review is incorrectly listed, please copy-paste <!--meta-tag:bot-skip--> into the comment that the bot should ignore.

Conflicts

Reviewers, this pull request conflicts with the following ones:

  • #33034 (wallet: Store transactions in a separate sqlite table by achow101)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

LLM Linter (✨ experimental)

Possible places where comparison-specific test macros should replace generic comparisons:

  • src/wallet/test/db_tests.cpp BOOST_CHECK_THROW(database.Open(), std::runtime_error); -> consider BOOST_CHECK_EXCEPTION with a message matcher so the test verifies the specific failure reason, not just the exception type.

2026-07-08 22:04:25

Comment thread test/functional/wallet_exported_watchonly.py Outdated
@pablomartin4btc pablomartin4btc force-pushed the wallet/exportwatchonly-inmemory-tmp-wallet branch from ebfee85 to 532c677 Compare July 4, 2026 11:35
@Sjors

Sjors commented Jul 4, 2026

Copy link
Copy Markdown
Member

Concept ACK

Code review 532c677.

My LLM friend found a potential issue: when TxnAbort() fails, SQLiteBatch::Close() tries to reopen the database, but it forgot about the SQLITE_OPEN_MEMORY flag. This is a problem in general, and can cause a crash here. It's worth fixing in a separate commit. E.g. you could have the SQLiteDatabase constructor store m_additional_flags.

Further more, consider the following patch:

  • consistently use <in-memory> in log messages
  • avoid using the display-only Filename() for file operations
    • maybe rename it to DisplayFileName()
  • improves migration code to also delete journal files before hitting assert(false)
diff --git a/src/wallet/sqlite.h b/src/wallet/sqlite.h
index 33c9628c49..4709967d39 100644
--- a/src/wallet/sqlite.h
+++ b/src/wallet/sqlite.h
@@ -178,5 +178,5 @@ class InMemoryWalletDatabase : public SQLiteDatabase
 public:
     InMemoryWalletDatabase();
-    std::string Filename() override { return "memory"; }
+    std::string Filename() override { return "<in-memory>"; }
     std::vector<fs::path> Files() override { return {}; }
 };
diff --git a/src/wallet/wallet.cpp b/src/wallet/wallet.cpp
index be618efbd6..6714b4ad95 100644
--- a/src/wallet/wallet.cpp
+++ b/src/wallet/wallet.cpp
@@ -3882,8 +3882,8 @@ bool CWallet::MigrateToSQLite(bilingual_str& error)
     }

-    // Close this database and delete the file
-    fs::path db_path = fs::PathFromString(m_database->Filename());
+    // Close this database and delete its file(s) - just one for BerkeleyRO
+    const std::vector<fs::path> old_files = m_database->Files();
     m_database->Close();
-    fs::remove(db_path);
+    for (const fs::path& file : old_files) fs::remove(file);

     // Generate the path for the location of the migrated wallet
@@ -3909,5 +3909,5 @@ bool CWallet::MigrateToSQLite(bilingual_str& error)
             batch->TxnAbort();
             m_database->Close();
-            fs::remove(m_database->Filename());
+            for (const fs::path& file : m_database->Files()) fs::remove(file);
             assert(false); // This is a critical error, the new db could not be written to. The original db exists as a backup, but we should not continue execution.
         }
@@ -4418,5 +4418,7 @@ util::Result<MigrationResult> MigrateLegacyToDescriptor(std::shared_ptr<CWallet>
             // Wallets stored directly as files in the top-level directory
             // (e.g. default unnamed wallets) don’t have a removable parent directory.
-            wallet_empty_dirs_to_remove.insert(fs::PathFromString(wallet.GetDatabase().Filename()).parent_path());
+            if (!files.empty()) {
+                wallet_empty_dirs_to_remove.insert(files.front().parent_path());
+            }
         }
     };

@pablomartin4btc

Copy link
Copy Markdown
Member Author

... a potential issue: when TxnAbort() fails, SQLiteBatch::Close() tries to reopen the database, but it forgot about the SQLITE_OPEN_MEMORY flag. This is a problem in general, and can cause a crash here. It's worth fixing in a separate commit. E.g. you could have the SQLiteDatabase constructor store m_additional_flags.

I'll add a fix for that on a separate commit (first/ base) as suggested, thanks!

Further more, consider the following patch:

  • consistently use <in-memory> in log messages

I agree, will change it.

  • avoid using the display-only Filename() for file operations

    • maybe rename it to DisplayFileName()

I agree with this also, but I'll do on a separate PR because I found that there are even more places to change (everything is within src/wallet/):

  • 1 pure virtual declaration: db.h:156
  • 3 overrides: sqlite.h:153, test/util.h:66, migrate.h:52
  • 5 display callers:
    • sqlite.cpp:56 — LogTrace log message
    • sqlite.cpp:278 — LogWarning log message
    • wallet.cpp:2386 — used in error/log messages after load
    • wallet.cpp:3105 (CreateNew) — logs "Wallet completed creation in %dms at %s"
    • wallet.cpp:3160 (LoadExisting) — same pattern
  • 4 file op callers: wallet.cpp:3885, wallet.cpp:3911, wallet.cpp:4420, export.cpp:93 - the last one disappears with this PR, the remaining three are fixed in the commit below.
  • improves migration code to also delete journal files before hitting assert(false)

On the migration fixes (wallet.cpp:3885, 3911, 4420) — these are still there and are genuine pre-existing issues:

  • Line 3885/3887: closes the old BerkeleyDB and removes only the main file, potentially missing its journal;
  • Line 3911: closes the new SQLite db and removes only the main file, potentially leaving wallet.dat-journal behind before hitting assert(false).

Although they're independent of this PR, I think these are worth fixing and I'll add a 3rd commit with it. Thank you!

@pablomartin4btc pablomartin4btc force-pushed the wallet/exportwatchonly-inmemory-tmp-wallet branch from 532c677 to 0c276ad Compare July 4, 2026 19:06
@pablomartin4btc

Copy link
Copy Markdown
Member Author

-Updates:

  • Addressed @Sjors's feedback:
    • Added a fix for a potential issue during TxnAbort() failure (1st commit);
    • Updated InMemoryWalletDatabase()::Filename() to return <in-memory>;
    • Improved migration code to also delete journal files (3rd commit).
  • Updated PR's description to reflect the above.

pablomartin4btc added a commit to pablomartin4btc/bitcoin that referenced this pull request Jul 5, 2026
…abase

`WalletDatabase::Filename()` is documented as returning a path "for logs
and error messages", but several callers use it for file operations
(removal, path derivation), which is incorrect — they should use
`Files()` instead.

Renaming to `DisplayFileName()` makes the display-only intent explicit
at every call site, making it immediately clear when a caller is
misusing it for file operations.

This is a pure rename: 1 pure virtual declaration, 3 overrides
(`SQLiteDatabase`, `MigrateDatabase`, `MockableSQLiteDatabase`), and
all callers. Callers that currently misuse the result for file
operations are left in place; fixing those by switching to `Files()` is
a separate concern being addressed in bitcoin#35655.

No behavior change.

Suggested by Sjors during review of bitcoin#35655.

Co-Authored-By: Sjors Provoost <sjors@sprovoost.nl>
@pablomartin4btc

Copy link
Copy Markdown
Member Author

-Updates:

  • After reflection, I folded the Filename()DisplayFileName() rename (originally planned as a separate PR) into this one as a 4th commit. It lands more cleanly here immediately after the Files() fix — at that point all remaining callers are display-only, so the rename needs no caveats.
  • Updated PR's description accordingly.

@Sjors Sjors left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review b67d6b5

I think MockableSQLiteDatabase should inherit from InMemoryWalletDatabase. That should also let you introduce the new subclass in its own commit, before _wallet: use in-memory SQLite for temporary wallet in exportwatchonlywallet_.

In 7b6136e wallet: store m_additional_flags in SQLiteDatabase to fix reopen path: there's a second problem that we should fix: it makes no sense to Close() and Open() an in-memory database, that's just going to wipe everything. It could throw() instead.

In b67d6b5 wallet, refactor: rename Filename() to DisplayFileName() in WalletDatabase: I think we should go one step further and set m_display_file_name at construction time. That's safer because the SQLiteDatabase constructor logs things, so we shouldn't rely on an override DisplayFileName(). It also ties into the next suggestions.

Note that the Files() helper, introduced as part of #31423, is just guessing which files exist. You could make that a bit more robust by using sqlite3_filename_journal in modern sqlite3 versions.

We also rely on InMemoryWalletDatabase to override Files() with an empty list. Instead we could make m_file_path and m_journal_file_path optional, and not set them for an in-memory wallet. I have a commit that does this, along with some constructor ergonomics.

Here's a branch that does all the above (except for splitting b67d6b5).

Comment thread src/wallet/sqlite.cpp Outdated
Comment thread src/wallet/wallet.cpp Outdated
Comment thread src/wallet/sqlite.cpp

@janb84 janb84 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Concept ACK b67d6b5

PR Uses the functionality of SQLite to create a in-memory database to use as the temporary storage func. of the WatchOnlyWallet. This has the advantage that it's really temporary and removes a lot of removal logic from the code. Which I think is an improvement.

NIT: The changed open function with stored additional flags can use a regression test (imho)

eg:

Details

something like this, as additional test in db_tests.cpp:

BOOST_AUTO_TEST_CASE(in_memory_connection_refresh)
{
    // Regression test for the reopen path of an in-memory database.
    InMemoryWalletDatabase database;

    std::string key = "key";
    std::string value = "value";

    // Reopening an in-memory database must not attempt any on-disk access.
    database.Close();
    BOOST_CHECK_NO_THROW(database.Open());

    // The refreshed in-memory connection must remain usable for reads and writes.
    SQLiteBatch batch{database};
    BOOST_CHECK(batch.Write(key, value));
    std::string read_value;
    BOOST_CHECK(batch.Read(key, read_value));
    BOOST_CHECK_EQUAL(read_value, value);
}

EDIT: i Agree with Sjors that throwing would be a better solution that recovering to a empty state.

Comment thread src/wallet/wallet.cpp Outdated
Comment thread src/wallet/export.cpp

@achow101 achow101 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Filename() is documented as returning a path for display/logging only, but the file-op callers above made its misuse invisible. After switching those to Files(), Filename() is renamed to DisplayFileName() to enforce the display-only intent at every remaining call site.

I think this is a mischaracterization of Filename(). When the documentation was originally written, it's only use was for logging, where we want to log the full path to the data file. Just because the documentation still says that does not mean that it is exclusively to be used for logging, nor that using it outside of logging is misuse. Filename() must point to the primary data file and callers can assume that it will be a full path to that file. I don't think this should be changed at all.

b67d6b5 "wallet, refactor: rename Filename() to DisplayFileName() in WalletDatabase" Should be dropped. I'm inclined to say that 0c276ad "wallet: use Files() instead of Filename() for file removal in migration" should be dropped as well. Both of these are also mostly unrelated to the main change in this PR.

Comment thread src/wallet/wallet.cpp Outdated
Comment thread src/wallet/wallet.cpp Outdated
@achow101

achow101 commented Jul 6, 2026

Copy link
Copy Markdown
Member

We also rely on InMemoryWalletDatabase to override Files() with an empty list. Instead we could make m_file_path and m_journal_file_path optional, and not set them for an in-memory wallet. I have a commit that does this, along with some constructor ergonomics.

No, every caller needs to intentionally choose what to put for the paths and names as these are relevant to sqlite's usage. We should not be defaulting names or paths as they may result in the wrong database files being opened.

@pablomartin4btc pablomartin4btc force-pushed the wallet/exportwatchonly-inmemory-tmp-wallet branch from b67d6b5 to 176797a Compare July 7, 2026 02:32
@pablomartin4btc

Copy link
Copy Markdown
Member Author

Code review b67d6b5

Done the first two — MockableSQLiteDatabase now derives from InMemoryWalletDatabase (removing its redundant Files() override), and the force_conn_refresh path now throws for in-memory connections.

On m_display_file_name and optional m_file_path: dropping commits 3 and 4 (per @achow101 's feedback) makes these unnecessary for this PR. Keeping Filename() as-is with the in-memory override, and keeping the explicit Files() override in InMemoryWalletDatabase.

@pablomartin4btc pablomartin4btc force-pushed the wallet/exportwatchonly-inmemory-tmp-wallet branch from 176797a to ce48d55 Compare July 7, 2026 03:23
@pablomartin4btc

Copy link
Copy Markdown
Member Author

Concept ACK b67d6b5

Thanks for the ACK and the regression test suggestion! Added it in a separate commit, co-authored by you. Since the behaviour changed from silently reopening to throwing, the assertion is inverted: BOOST_CHECK_THROW(database.Open(), std::runtime_error) instead of BOOST_CHECK_NO_THROW.

@pablomartin4btc

Copy link
Copy Markdown
Member Author

We also rely on InMemoryWalletDatabase to override Files() with an empty list. Instead we could make m_file_path and m_journal_file_path optional, and not set them for an in-memory wallet. I have a commit that does this, along with some constructor ergonomics.

No, every caller needs to intentionally choose what to put for the paths and names as these are relevant to sqlite's usage. We should not be defaulting names or paths as they may result in the wrong database files being opened.

Agreed, keeping the explicit Files() override in InMemoryWalletDatabase.

@pablomartin4btc

Copy link
Copy Markdown
Member Author

b67d6b5 "wallet, refactor: rename Filename() to DisplayFileName() in WalletDatabase" Should be dropped. I'm inclined to say that 0c276ad "wallet: use Files() instead of Filename() for file removal in migration" should be dropped as well.

Both dropped, thanks.

@pablomartin4btc

Copy link
Copy Markdown
Member Author

-Updates:

  • Addressed @achow101's feedback: dropped old commits 3 and 4 (Files() migration fix and DisplayFileName() rename) — both unrelated to the main change.
  • Addressed @Sjors's feedback: m_additional_flags stored as member in commit 1; force_conn_refresh path and public Open() now throw for in-memory databases; added friend class SQLiteBatch; MockableSQLiteDatabase now derives from InMemoryWalletDatabase in commit 2, removing its redundant Files() override.
  • Addressed @janb84's feedback: regression test added as a separate commit 3.
  • Updated PR description accordingly.

@DrahtBot DrahtBot removed the CI failed label Jul 7, 2026
@Sjors

Sjors commented Jul 7, 2026

Copy link
Copy Markdown
Member

@achow101 wrote:

Filename() must point to the primary data file and callers can assume that it will be a full path to that file. I don't think this should be changed at all.

Then let's actually document the function to say so. But it's inconsistent with then setting it to "<in-memory>" in a subclass. So perhaps it still makes sense to make it an optional, something along the lines of what I did in Sjors@69b5f28, but leaving Files() alone.

We also rely on InMemoryWalletDatabase to override Files() with an empty list. Instead we could make m_file_path and m_journal_file_path optional, and not set them for an in-memory wallet. I have a commit that does this, along with some constructor ergonomics.

No, every caller needs to intentionally choose what to put for the paths and names as these are relevant to sqlite's usage. We should not be defaulting names or paths as they may result in the wrong database files being opened.

What do you mean by this? m_file_path is set by the constructor, Sjors@69b5f28 doesn't change that. It just avoids setting a fake filename for an in-memory wallet.

@achow101

achow101 commented Jul 7, 2026

Copy link
Copy Markdown
Member

No, every caller needs to intentionally choose what to put for the paths and names as these are relevant to sqlite's usage. We should not be defaulting names or paths as they may result in the wrong database files being opened.

What do you mean by this? m_file_path is set by the constructor, Sjors@69b5f28 doesn't change that. It just avoids setting a fake filename for an in-memory wallet.

You're literally allowing file path to be nullopt, and then choosing an default when it is, and passing that default in to sqlite_open_v2. That is unsafe, as it implies that callers do not need to pass a file path, when in fact they must in order to use sqlite_open_v2 correctly. The path given has a meaning to sqlite, callers must know that it does and we should not be choosing a default value because that is how we end up accidentally using the wrong database.

If a subclass wants to use a bogus file path, it must do so explicitly by choice. SQLiteDatabase must not be choosing a default for it because that hides what the file path really is.

Comment thread src/wallet/export.cpp Outdated
Comment thread src/wallet/sqlite.cpp Outdated
@pablomartin4btc pablomartin4btc force-pushed the wallet/exportwatchonly-inmemory-tmp-wallet branch 2 times, most recently from 08ca2a5 to 3165ed8 Compare July 8, 2026 00:18
@pablomartin4btc

Copy link
Copy Markdown
Member Author

-Updates:

  • Wallet temp name is now "<source_wallet>_watchonly_temp" (derived from the source wallet) rather than the fixed "exportwatchonly_temp", to avoid naming conflicts when exporting multiple wallets concurrently and to keep log lines traceable to the source — per @achow101.
  • InMemoryWalletDatabase now passes ":memory:" as file_path to sqlite3_open_v2 (the canonical SQLite name for in-memory databases) rather than an empty string, which has a different meaning in SQLite (temporary file on disk). The Filename() override is removed — the base class now returns ":memory:" directly — per @achow101.

@DrahtBot DrahtBot removed the CI failed label Jul 8, 2026
@Sjors

Sjors commented Jul 8, 2026

Copy link
Copy Markdown
Member

utACK 3165ed8

@achow101 thanks for the clarification.

What do you mean by this? m_file_path is set by the constructor, Sjors@69b5f28 doesn't change that. It just avoids setting a fake filename for an in-memory wallet.

You're literally allowing file path to be nullopt, and then choosing an default when it is, and passing that default in to sqlite_open_v2

The default should have been the magic string :memory: as you pointed out here: #35655 (comment)

The path given has a meaning to sqlite, callers must know that it does and we should not be choosing a default value because that is how we end up accidentally using the wrong database.

The commit enforces Assert(m_file_path.has_value() != bool(m_additional_flags & SQLITE_OPEN_MEMORY));. So when the database has a real path the caller must provide it.

The fact that sqlite_open_v2 uses a magic file name doesn't need to bleed through to higher abstractions, especially since we need to decide whether to delete files.

Latest version of what I had in mind: Sjors@7c6026c

That said, I think the Files() override { return {} } hack is fine for now.

@DrahtBot DrahtBot requested a review from janb84 July 8, 2026 10:16
@pablomartin4btc

Copy link
Copy Markdown
Member Author

Latest version of what I had in mind: Sjors@7c6026c

Thanks for working on that idea, I find it interesting and I like the approach. I noticed some details (e.g. magic string I think should live and belongs to InMemoryWalletDatabase class)) I'd prefer to review more carefully in a follow-up rather than bundling here. Happy to revisit in a follow-up if you want to explore it further, but I'd prefer to keep SQLiteDatabase core unchanged for now.

Comment thread src/wallet/sqlite.cpp Outdated
SQLiteDatabase::Open() (the public override) always reopens the database
with no additional flags. If SQLiteBatch::Close() triggers the
force_conn_refresh path (TxnAbort failed), it calls Open() which drops
the original additional_flags, causing in-memory databases to be reopened
as on-disk instead.

Store additional_flags as a member and use it in Open() so the reconnect
preserves the original flags. For in-memory databases, connection recovery
makes no sense as all data would be lost; both the force_conn_refresh path
and the public Open() now throw instead.

Co-authored-by: Sjors Provoost <sjors@sprovoost.nl>
@pablomartin4btc pablomartin4btc force-pushed the wallet/exportwatchonly-inmemory-tmp-wallet branch from 3165ed8 to 66d4d97 Compare July 8, 2026 22:00
pablomartin4btc and others added 2 commits July 8, 2026 19:03
…allet

The intermediate watchonly wallet created during exportwatchonlywallet is
a pure build artifact — it is always discarded once BackupWallet() copies
it to the destination. Creating it as an in-memory SQLiteDatabase
(SQLITE_OPEN_MEMORY) removes the need to write files to the wallets
directory and eliminates the cleanup handler that deleted those files on
both success and failure paths.

Introduces InMemoryWalletDatabase (a minimal SQLiteDatabase subclass) and
MakeInMemoryWalletDatabase() factory in sqlite.h/cpp, following the same
pattern as MockableSQLiteDatabase / CreateMockableWalletDatabase() in the
test utilities. MockableSQLiteDatabase now derives from InMemoryWalletDatabase,
removing its redundant Files() override.

The wallet is named after the source wallet ("<name>_watchonly_temp") so
concurrent exports of different wallets use distinct names and log lines
remain traceable to the source wallet.
InMemoryWalletDatabase::Open() now throws to prevent silently returning
a fresh empty connection after close, which would discard all data. Add a
test to pin this behaviour.

Co-authored-by: Jan B <608446+janb84@users.noreply.github.com>
@pablomartin4btc pablomartin4btc force-pushed the wallet/exportwatchonly-inmemory-tmp-wallet branch from 66d4d97 to 777d23f Compare July 8, 2026 22:04
@Sjors

Sjors commented Jul 9, 2026

Copy link
Copy Markdown
Member

re-utACK 777d23f

Just a text change since my previous review.

I'd prefer to keep SQLiteDatabase core unchanged for now.

Yes, that's fine.

@achow101

achow101 commented Jul 9, 2026

Copy link
Copy Markdown
Member

ACK 777d23f

@janb84 janb84 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK 777d23f

Thanks for taking my suggestion, think this is sufficient for now although I hope to see a follow-up with the suggestions of @Sjors, given it's a bit more elegant (imho)

@achow101 achow101 merged commit ef101b0 into bitcoin:master Jul 10, 2026
28 checks passed
Kino1994 pushed a commit to Kino1994/bitcoin-full-history that referenced this pull request Jul 12, 2026
…ry wallet in exportwatchonlywallet

5a96701 test: add regression test for in-memory SQLiteDatabase reopen (Pablo Martin)
7b90483 wallet: use in-memory SQLite for temporary wallet in exportwatchonlywallet (Pablo Martin)
194403b wallet: store m_additional_flags in SQLiteDatabase to fix reopen path (Pablo Martin)

Pull request description:

  Since #33032 landed (in-memory `SQLiteDatabase` via `SQLITE_OPEN_MEMORY`), the intermediate wallet built during `exportwatchonlywallet` can live entirely in memory instead of being written to the wallets directory as a temporary file.

  The temp wallet is a pure build artifact: it is populated with descriptors, transactions, and address book data, then immediately discarded once `BackupWallet()` copies its contents to the destination file. Making it in-memory removes all on-disk footprint and eliminates the `cleanup_watchonly_wallet` RAII handler — along with the `wallet_path` and `cleanup_files` variables it needed — which previously ensured the temp files were deleted on both success and failure paths.

  This PR introduces `InMemoryWalletDatabase` (a minimal `SQLiteDatabase` subclass) and `MakeInMemoryWalletDatabase()` factory in `sqlite.h/cpp`, following the same pattern as `MockableSQLiteDatabase` / `CreateMockableWalletDatabase()`. `MockableSQLiteDatabase` now derives from `InMemoryWalletDatabase`, removing its redundant `Files()` override.

  Suggested by Sjors in #32489 ([comment](bitcoin/bitcoin#32489 (comment))).

  ---
  Also fixes a related issue found (by Sjors) during review:

  - `SQLiteDatabase::Open()` (the no-arg public override) hardcoded 0 as `additional_flags` when reopening after a failed `TxnAbort()`, which would reopen an in-memory database as on-disk. Fixed by storing `m_additional_flags` in the constructor and using it in the reopen path. For in-memory databases, both the `force_conn_refresh` path and the public `Open()` now throw instead of silently creating a fresh empty connection. A regression test for the `Open()` throw is included in a separate commit.

  ---
  As a follow-up, `InMemoryWalletDatabase` could replace `MockableSQLiteDatabase` in `src/bench/` (5 files, 6 call sites), since benchmarks don't need mock-specific behaviour and benefit from using the same in-memory path as production code.

ACKs for top commit:
  Sjors:
    re-utACK 5a96701
  achow101:
    ACK 5a96701
  janb84:
    ACK 5a96701

Tree-SHA512: 71178ce99c7ebc0fc5ba17956c27d37f90e3d36cefbdc0d15d1424b7a70bf15f05a13cb03c268d885678aba1fd3c90567d8389d3680650c8ae46f4cb4b10b28a
pablomartin4btc added a commit to pablomartin4btc/bitcoin that referenced this pull request Jul 13, 2026
…tabase

Benchmarks don't need mock-specific behaviour (overridden Filename(),
Format(), or the exposed batch-level WriteKey()). Replace
CreateMockableWalletDatabase() with MakeInMemoryWalletDatabase() across
6 call sites in src/bench/ (5 files), using the same in-memory SQLite
path that production code uses.

wallet_migration.cpp is excluded: it calls GetOrCreateLegacyDataSPKM()
which asserts Format() == "sqlite-mock", a signal used to allow legacy
SPKM setup in test/bench contexts. MockableSQLiteDatabase is still
correct there.

For coin_selection.cpp, which had no other dependencies on
wallet/test/util.h, also switch the include to <wallet/sqlite.h>.

Follow-up suggested in bitcoin#35655.
pablomartin4btc added a commit to pablomartin4btc/bitcoin that referenced this pull request Jul 13, 2026
…tabase

Benchmarks don't need mock-specific behaviour (overridden Filename(),
Format(), or the exposed batch-level WriteKey()). Replace
CreateMockableWalletDatabase() with MakeInMemoryWalletDatabase() across
6 call sites in src/bench/ (5 files), using the same in-memory SQLite
path that production code uses.

wallet_migration.cpp is excluded: it calls GetOrCreateLegacyDataSPKM()
which asserts Format() == "sqlite-mock", a signal used to allow legacy
SPKM setup in test/bench contexts. MockableSQLiteDatabase is still
correct there.

For coin_selection.cpp, which had no other dependencies on
wallet/test/util.h, also switch the include to <wallet/sqlite.h>.

Follow-up suggested in bitcoin#35655.
sedited added a commit that referenced this pull request Jul 13, 2026
…MemoryWalletDatabase

7508ac3 bench: replace CreateMockableWalletDatabase with MakeInMemoryWalletDatabase (Pablo Martin)

Pull request description:

  Benchmarks don't need mock-specific behaviour (overridden `Filename()`, `Format()`, or the exposed batch-level `WriteKey()`). Replace `CreateMockableWalletDatabase()` with `MakeInMemoryWalletDatabase()` across all 6 call sites in `src/bench/` (5 files - 4 wallet bench files + 1 in coin_selection), using the same in-memory SQLite path that production code uses.

  `wallet_migration.cpp` is excluded: it calls `GetOrCreateLegacyDataSPKM()` which asserts `Format() == "sqlite-mock"`, a deliberate signal that allows legacy SPKM setup in `test/bench` contexts. `MockableSQLiteDatabase` is still correct there.

  For `coin_selection.cpp`, which had no other dependencies on `wallet/test/util.h`, the include is switched to `<wallet/sqlite.h>`. The remaining 4 files retain `wallet/test/util.h` for other utilities but also add an explicit `<wallet/sqlite.h>` include as required by IWYU.

  Follow-up suggested in #35655.

ACKs for top commit:
  sedited:
    ACK 7508ac3
  janb84:
    ACK 7508ac3

Tree-SHA512: 6f4086eb5700ba3da882378dcee76b3c075670cdea5d71b45c9b369e48c72b8f591e0c5a6bd4504719dd512b3b7d2b23230a8b76e471c6c0571c13325fa3cba0
alexanderwiederin added a commit to alexanderwiederin/rust-bitcoinkernel that referenced this pull request Jul 14, 2026
…4e48e968a

114e48e968a kernel: Add script tracer
54e1a95a12e Merge bitcoin/bitcoin#35719: ci: disable Qt build in OpenBSD cross job
b0e09511586 ci: disable Qt build in OpenBSD cross job
734c34bafda Merge bitcoin/bitcoin#35427: depends: Build `qt` and `qrencode` packages on OpenBSD
ee61b11a9e7 Merge bitcoin/bitcoin#35200: node: smooth oversized `dbcache` warnings
e3554bf361f Merge bitcoin/bitcoin#35579: wallet: reserve walletrescan before checking wallet is at the tip
11ae4265522 Merge bitcoin/bitcoin#35715: cmake: Fix WITH_EXTERNAL_LIBMULTIPROCESS + BUILD_FUZZ_BINARY
e544413c0de Merge bitcoin/bitcoin#32763: wallet: Replace CWalletTx::mapValue and vOrderForm with explicit class members
fe1cb6e40d7 Merge bitcoin/bitcoin#35690: wallet: Introduce WalletError with machine-readable error code
441f3114f57 Merge bitcoin/bitcoin#35659: Clarify supported *BSD releases and drop outdated workarounds
0399df827c6 Merge bitcoin/bitcoin#35708: depends: capnp 1.5.0
1ab1fdd4696 Merge bitcoin/bitcoin#35705: bench: replace CreateMockableWalletDatabase with MakeInMemoryWalletDatabase
db35b9238fc ipc # build: Fix fuzz target CMakeLists.txt for external libmultiprocess
d18fec892e2 Merge bitcoin/bitcoin#35698: doc: Update enum class constant naming style guide
a2e4cd7ad2a depends: capnp 1.5.0
7508ac319d9 bench: replace CreateMockableWalletDatabase with MakeInMemoryWalletDatabase
907e284e303 Merge bitcoin/bitcoin#35701: test: Remove `mock_process.cpp`
c8459b6bdcd Merge bitcoin/bitcoin#35568: txospenderindex: disable bloom filters to optimize disk usage
63c5f9d22c0 test: Remove `mock_process.cpp`
ef101b04a8d Merge bitcoin/bitcoin#35655: wallet: Use in-memory SQLite for temporary wallet in exportwatchonlywallet
b6becf3534c Merge bitcoin/bitcoin#35684: Update libmultiprocess subtree to add `max_connections` option
930f25050f7 Merge bitcoin/bitcoin#35700: doc: archive release notes for v29.4
9b2b3f4ec6f doc: archive release notes for v29.4
297fd1489bb Merge bitcoin/bitcoin#35412: ci: add NetBSD Clang cross job
e314869066b Merge bitcoin/bitcoin#35695: Remove myself as security contact
e3d67a5eae5 Merge bitcoin/bitcoin#35691: chainparams: delete my DNS seed
a8223bb4e62 wallet: Introduce WalletError with machine-readable error code
fad5809cb92 doc: Update enum class constant naming style guide
7d8137c1417 Merge bitcoin/bitcoin#34897: indexes: Don't commit ahead of the flushed chainstate
629df81e4c6 Remove myself as security contact
d164a043426 node: smooth oversized `-dbcache` warnings
d9080639804 chainparams: delete my DNS seed
e9ed898a0da validation: Don't use m_chain.Tip() in FlushStateToDisk
3679f1ecf5e index: Don't commit ahead of the flushed chainstate
65735728a5a index: Remove return value from Commit()
09c06960c6a validation: track last flushed block
13c02b5466d test: add test for index commits ahead of the last flushed block
c43b7a1115f ci: add netBSD cross CI job
699c21aea47 depends: add netbsd_LDFLAGS
777d23f25c7 test: add regression test for in-memory SQLiteDatabase reopen
d1e7f8c986f wallet: use in-memory SQLite for temporary wallet in exportwatchonlywallet
ee43743f126 wallet: store m_additional_flags in SQLiteDatabase to fix reopen path
a9d1b652f32 Merge commit '707d0ded84563386f770ec17970834c65f8fa938' into pr/subtree-12
707d0ded845 Squashed 'src/ipc/libmultiprocess/' changes from 16bf05dea02..28e056576a3
2bab6bc73f2 refactor: Drop support for FreeBSD < 14
91b5c8a07c6 refactor: Remove FreeBSD-specific workaround
56701ff6d5c doc: Clarify supported *BSD releases
6d0ea4cf5bd doc: add release notes
a2b1c86903a txospenderindex: disable bloom filters to optimize disk usage
fed3cf6f0ed wallet: Replace CWalletTx's vOrderForm with specific fields
4f8823e8e11 wallet: Drop vOrderForm from CommitTransaction
9e62e4b1f34 test: slow down rescaning process
a2b0bfcd854 wallet: Drop mapValue from CWalletTx
cb99864c91c wallet: Throw if unknown entry is found in mapValue
98d5cdae663 wallet: Make CWalletTx "replaces_txid" and "replaced_by_txid" member variables
7ef8a6efc2a wallet: Make CWalletTx "comment" and "to" member variables
2155e913d3e wallet: Make CWalletTx "from" and "message" member variables
c6ba98dcc8a wallet: Drop mapValue from CommitTransaction
00abb174a80 wallet: Pass comment and comment_to to CommitTransaction
1a219a37a21 wallet: Pass replaces_txid to CommitTransaction outside of mapValue
336f5a738b3 wallet: reserve walletrescan before checking wallet is at the tip
a54ec373a69 depends: Build `qt` and `qrencode` packages for OpenBSD hosts

git-subtree-dir: libbitcoinkernel-sys/bitcoin
git-subtree-split: 114e48e968a087f74c1ab611ac2a31a9266812e1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants