Skip to content

release: LND 0.21.0-beta — SQLite backend, gRPC/REST TLS passthrough, native onion messages#165

Draft
helix-nine wants to merge 16 commits into
masterfrom
next
Draft

release: LND 0.21.0-beta — SQLite backend, gRPC/REST TLS passthrough, native onion messages#165
helix-nine wants to merge 16 commits into
masterfrom
next

Conversation

@helix-nine

@helix-nine helix-nine commented Jun 6, 2026

Copy link
Copy Markdown
Contributor

LND 0.21.0-beta release (nextmaster)

The full 0.21 release, now consolidated on next:

  • LND 0.21.0-beta upgrade.
  • SQLite database backend — one-time, on-upgrade bolt→SQLite migration via lndinit plus native-SQL conversion. Irreversible; release notes warn to back up first. Verified end-to-end on a real bitcoind node (137 MB channel.db): schema-finalize → lndinit migrate-db → node back up, synced, wallet intact.
  • gRPC + REST TLS passthrough — LND terminates its own TLS end-to-end (the StartOS-issued cert reaches the client); the REST lndconnect URL now carries the cert, so wallets like Zeus connect reliably over Tor and LAN.
  • Native onion messages (BOLT12) — 0.21 advertises feature bit 39 natively, so the obsolete custom-protocol toggle is removed and stale custom-init/nodeann=39 are stripped on upgrade (they otherwise crash 0.21 with feature bit: 39 already set).

⚠️ DRAFT — blocked on upstream

TODO: waiting on Lightning Labs to publish the final lightninglabs/lndinit:v0.1.35-beta-lnd-v0.21.0-beta image. The Dockerfile pins the lndinit binary from the .rc3 release-candidate image because no final …-lnd-v0.21.0-beta lndinit pairing exists yet. Before merge/release: swap the COPY source in Dockerfile to the final tag (drop .rc3) — see the TODO(release) comment there.

Coordinated downstream change

bolt12-pay must drop its LND autoconfig task (onion messages are native now) and bump its lnd versionRange to this release — tracked separately.

helix-nine and others added 2 commits June 6, 2026 15:38
Move LND to the SQLite backend with native SQL. Existing bolt nodes (and
Umbrel / pre-0.21 StartOS imports) are converted on first start by a
migrate-sqlite oneshot that gates the lnd daemon; a Database Migration health
check shows progress. Fresh installs are born on SQLite.

- enforce db.backend=sqlite + db.use-native-sql=true in lnd.conf
- migrate-sqlite oneshot: run lnd once on bolt to finalize the kv schema, then
  lndinit migrate-db to copy buckets to SQLite; finalize run overrides
  --db.backend=bolt --db.use-native-sql=false on the CLI
- track progress in startup-flags.json (no .const-watched writes -> no restart
  on completion); oneshot + health check are absent on non-migrating starts
- build image from ./Dockerfile (lightninglabs/lnd + lndinit binary)
- also: default tor.skip-proxy-for-clearnet-targets to true in the model and
  drop its install seed (completes the pending TODO)
- update README + instructions

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Bind both ports as passthrough (protocol/addSsl null + secure.ssl) so
LND's own StartOS-issued cert reaches the client, instead of StartOS
terminating TLS at the edge with the device cert and breaking clients
that pin LND's cert via the lndconnect URL. protocol 'https' does not do
this — the SDK still synthesizes an addSsl config for it.

Add the cert to the REST lndconnect query so REST clients can pin it the
same way gRPC clients do, and update the release notes.
…ion-messages config

The 0.21 bump + SQLite backend never actually ran; found and fixed via an
end-to-end test on a real bitcoind node:

- native SQL is set on the lnd daemon CLI, not the conf — a conf-level bool
  can't be turned off on the CLI, so the bolt schema-finalize run failed
  ("cannot use native SQL with bolt backend"). The schema renders it out.
- restore curl in the image: LND 0.21 switched to Alpine and dropped it,
  breaking every REST call we shell out for, including wallet unlock.
- supply --fee.url for the neutrino-override schema-finalize run, so it starts
  cleanly on bitcoind-backed nodes (neutrino on mainnet requires it).
- rely on lndinit's idempotent exit code for resume (re-running migrate-db on a
  tombstoned source exits 0); drop the best-effort substring guess.
- db-migration health check shows its loading message (statusTrigger needs a
  short `starting` interval, else the first poll waits the 10-min default).

Onion messages are native in LND 0.21 (feature bit 39), so the pre-0.21 custom
protocol entries (custom-init/nodeann=39) are obsolete and now crash the node
("feature bit 39 already set"). Remove the onion-messages config toggle and its
form mapping, and force custom-init/nodeann/message to undefined so the schema
strips them from upgraded confs. bolt12-pay drops its autoconfig task and bumps
its lnd versionRange separately.

Comprehensive release notes for 0.21.0-beta:0.
@MattDHill MattDHill marked this pull request as draft June 13, 2026 01:11
@MattDHill MattDHill changed the title chore: bump LND to 0.21.0-beta; 0.20.1-beta:13 → 0.21.0-beta:0 release: LND 0.21.0-beta — SQLite backend, gRPC/REST TLS passthrough, native onion messages Jun 13, 2026
MattDHill and others added 2 commits June 16, 2026 14:15
Lightning Labs never published a final v0.1.35-beta-lnd-v0.21.0-beta pairing;
it went straight from the .rc3 tag to bumping lndinit to v0.1.36. v0.1.36-beta
is a final (non-rc) release paired with lnd v0.21.0-beta — newer than the
required v0.1.35, so it carries the same channeldb-schema-v35 support the
bolt → SQLite migration needs. Drops the dependency on an RC tag and resolves
the TODO.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Patch release: Tor v3 onion creation fix, DNS fallback crash fix, on-chain forward-interceptor settlement fix, and stricter final-hop CLTV-expiry validation. Bumps the lnd and lndinit images to v0.21.1-beta.
@MattDHill MattDHill force-pushed the next branch 2 times, most recently from 48c8f22 to 5ed5b95 Compare July 4, 2026 01:08
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F5qZh4QiVySBLkV5ya7XSr
MattDHill and others added 3 commits July 6, 2026 09:41
…n real address changes

Doctrine v3 (issue start-technologies#3407): dependency addresses go
through a utils.ts bridgeAddress helper chained .const() — the mapped
value is the minimal bridge address, so main never restarts on
dependency updates, and heals with one restart when a dependency is
installed after this service, uninstalled, or changes port. Cross-
package container IPs and .startos dials are gone; static config
defaults become loopback placeholders; tor consumers fold the
allocator-guaranteed 9050 fallback into the map.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PpJxhk25X42WTafMdkUx6t
- no fabricated dependency addresses: absent dependency => absent
  config/env value (optional fields, omitted keys), never a fake
  127.0.0.1 dial default
- dependency-task accept entries match absent-when-default fields
  (prune: 0 or absent both accepted)
- neutral naming: the Bitcoin dependency is 'Bitcoin', not 'Bitcoin
  Core'/'Bitcoin Knots', across prose, release notes, and i18n
- instructions.md keeps no version/migration notes
- remove orphaned i18n entries added by the migration
- normalize the bridgeAddress helper to an explicit early-return and
  prettier-format wave-touched files

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PpJxhk25X42WTafMdkUx6t
Bumps @start9labs/start-sdk 2.0.1 -> 2.0.3 (dependency and lockfile only).

- 2.0.3 fixes a dependency-gated daemon wedging permanently after its
  dependency's readiness flaps: Daemon.term() unconditionally destroyed the
  daemon's SubContainer, so the next start() threw "already destroyed" and the
  daemon never recovered. Adds a non-destroying Daemon.stop() for
  dependency-driven pauses, and serializes pause/resume transitions.
- 2.0.2 runs s9pk.mk's check-deps before packing, not just before install.

No packaging API changed between 2.0.1 and 2.0.3, so no source changes were
needed. Verified with `npm run check` (tsc --noEmit).
@helix-nine

Copy link
Copy Markdown
Contributor Author

Moved the bolt→SQLite migration out of main and into an init step (startos/init/migrateSqlite.ts), driven by runUntilSuccess — pushed to next (7b79ea9).

What changed

  • startos/init/migrateSqlite.ts (new): setupOnInit gated on needsSqliteMigration(); when a conversion is pending it builds a one-oneshot sdk.Daemons.of(effects).addOneshot('migrate-sqlite', …).runUntilSuccess(…). Wired into init/index.ts right after setupCerts (the conversion curls a temporary LND on 127.0.0.1, and setupCerts is what writes the tls.cert covering that SAN).
  • main.ts: dropped the migrate-sqlite oneshot, the db-migration health check, and the needMigration gating. The lnd daemon no longer requires the migration. main is now migration-agnostic — by the time it runs, the node is already on SQLite.
  • The verified conversion logic is unchanged (runSqliteMigration → schema-finalize run on bolt + lndinit migrate-db). Only where it runs moved. Removed sqliteMigrationComplete (it only fed the deleted health check), pruned the now-dead Database Migration i18n strings across all locales, and refreshed the README migration section (dropped the health-check row).

Verified: tsc --noEmit clean under SDK 2.0.3, ncc build clean, make x86 packs a valid s9pk. (The migration itself needs a real pre-0.21 bolt node to exercise end-to-end — not reproducible in the build sandbox.)

Two things worth a look:

  1. Progress visibility. In main the conversion surfaced a Database Migration health check ("Migrating the database to SQLite… — do not interrupt"). As an init step it runs before the service starts, so the UI shows the generic initializing state instead — no "don't interrupt" message. The release notes already say the first start "may take a while," so I left them as-is, but if you want a user-facing cue during the conversion I can add a notification at migration start.
  2. runUntilSuccess shape. I kept it as a single oneshot (the "Oneshots Only… like database migrations" pattern from the recipe), so the temporary LND is still spawned/killed inside finalizeBoltSchema. The more idiomatic alternative is to let runUntilSuccess manage LND as a real daemon in the chain (daemon = schema-finalize LND, dependent oneshot = unlock + wait, then a second chain for lndinit), which would delete the manual sub.spawn/child.kill. I went with the lower-risk single-oneshot since I can't run the real migration here to re-validate a rewrite — happy to switch to the daemon-managed shape if you'd prefer it.

One behavioral note on failure handling: runUntilSuccess only surfaces a failed oneshot via its timeout (it doesn't fail fast), so I set a generous 60-min backstop. On a genuine error the conversion throws early and, being resumable, retries on the next init.

Move the conversion out of setupMain's daemon chain into the migrate init step,
and drive it the idiomatic way — two temporary runUntilSuccess daemon chains
instead of a hand-rolled sub.spawn/child.kill.

- init/migrateSqlite.ts: needsSqliteMigration() gate, then runSqliteMigration.
  Ordered after setupCerts (the conversion curls a temporary LND on 127.0.0.1,
  and setupCerts writes the tls.cert covering that SAN).
- sqliteBackend.ts:
  - finalizeBoltSchema runs LND as a managed daemon in a runUntilSuccess chain;
    a dependent oneshot unlocks the wallet and waits for UNLOCKED, then the SDK
    tears LND down. The chain owns its own subcontainer, which term() destroys —
    so lndinit later opens a fully-closed db.
  - lndinit migrate-db runs as a second runUntilSuccess oneshot in a fresh
    subcontainer (the first is destroyed on teardown).
  - drop the manual spawn/kill, the per-call waitForState timeout (the
    runUntilSuccess timeout is now the single bound), and sqliteMigrationComplete.
- main.ts: drop the migrate-sqlite oneshot, the db-migration health check, and
  the needMigration gating; the lnd daemon no longer requires the migration.
- remove the now-dead Database Migration i18n strings across all locales; update
  the README migration section and drop its health-check row.

tsc + ncc + `make x86` pass. End-to-end exercise needs a real pre-0.21 bolt node.
@helix-nine

Copy link
Copy Markdown
Contributor Author

Rewrote the migration to the idiomatic runUntilSuccess path — LND now runs as a managed daemon in the chain instead of the hand-rolled sub.spawn/child.kill. Force-pushed as one squashed commit (e89733c).

Design — two runUntilSuccess chains (it has to be two, because stage 2 needs LND stopped):

  1. finalize-schema — a daemon (lnd-schema: LND on bolt+neutrino, --db.backend=bolt) whose ready check passes once the wallet unlocker is serving (LOCKED), plus a dependent oneshot that unlocks the wallet and waits for UNLOCKED. LND applies the channeldb schema migrations during unlock, so UNLOCKED is the guarantee they're done. runUntilSuccess then tears LND down.
  2. lndinit-migrate — a oneshot running lndinit migrate-db, in a separate subcontainer, with LND fully stopped.

Why the design is shaped this way (from reading the SDK 2.0.3 source, since the old internals weren't trustworthy):

  • Daemon.term() awaits the process exiting and then destroys the subcontainer. So chain 1 fully stops LND before chain 2's lndinit opens the db (no half-open-db race) — but a term'd subcontainer is one-way destroyed, so each chain gets its own (lnd-schema / lnd-lndinit).
  • Daemons auto-restart on crash (backoff), and oneshots auto-retry on failure (only a successful exit breaks the loop). So a transient LND/unlock hiccup recovers inside the chain; the finalize oneshot is idempotent across retries (unlock tolerates "already unlocked", the wait is a poll). lndinit is idempotent too (tombstones, exits 0 on re-run).
  • defaultTrigger polls every 1s while "starting", so the ready gate fires promptly — no custom trigger needed.
  • restlisten is enforced to 0.0.0.0:8080, so the migration's 127.0.0.1:8080 REST calls are reachable.

Also dropped the per-call waitForState timeout — the runUntilSuccess timeout (30 min/stage, generous; resumable) is now the single bound.

Verified: tsc clean (SDK 2.0.3), ncc build clean, make x86 packs a valid s9pk.

Not verified end-to-end: exercising the actual conversion needs a real pre-0.21 bolt LND node with channel state, which I can't stand up in the build sandbox (old LND + wallet + mainnet channel.db). The mechanics follow the documented runUntilSuccess recipe and the SDK's teardown/retry semantics above, but if you want a live smoke test I'd need a bolt-format data set to point it at — let me know if you have one or want me to try building one on a VM.

@helix-nine

Copy link
Copy Markdown
Contributor Author

Tested the upgrade end-to-end — it works.

Installed the released 0.20.1-beta:13 (bolt) on a StartOS master VM, set backend=neutrino, initialized a wallet, and started it → bolt channel.db created (channeldb schema v33). Stopped it, then sideloaded this PR's build (0.21.1-beta:0, e89733c) as an upgrade. The migration ran during init exactly as designed:

Migrating 0.20.1-beta:13 -> 0.21.1-beta:0
Launching lnd-schema...                       # chain 1: LND-on-bolt daemon
CHDB: Checking for schema update: latest_version=35, db_version=35   # 33 -> 35 applied
Launching finalize-schema...                  # dependent oneshot
BTWL: The wallet has been unlocked without a time limit
Launching __RUN_UNTIL_SUCCESS...              # chain 1 sentinel -> LND torn down
Launching lndinit-migrate...                  # chain 2: LND stopped
MIGKV-channeldb: Migrating root bucket: open-chan-bucket   # ...all buckets...
LNDINIT !!!Migration of all mandatory db parts completed successfully!!!
LNDINIT Migrated DBs: [channeldb macaroondb decayedlogdb walletdb neutrinodb]
Initialization complete.

After the migration:

  • data/graph/mainnet/ now has channel.sqlite (+ -shm/-wal) and lnd.sqlite; the old channel.db is tombstoned with a channel.db.migrated-to-sqlite-… marker.
  • Started the service → LND 0.21.1-beta comes up on SQLite, wallet unlocks, lncli getinfo returns version: 0.21.1-beta and the node identity_pubkey (wallet intact).
  • Restarted again → the migration did not re-run (needsSqliteMigration() short-circuits once channel.sqlite is present / dbMigrationComplete is set) — straight to the normal main chain.

So the two runUntilSuccess chains behave as intended on a real bolt→SQLite upgrade: schema-finalize daemon → teardown → lndinit on the closed db → node up on SQLite. (Tested on neutrino to avoid needing a synced bitcoind; the conversion path is backend-agnostic.)

The init framework hands each handler a FullProgressTracker (3rd arg) that
auto-syncs to setInitProgress. Report the conversion as two named phases so the
StartOS update UI shows what's happening instead of a bare "initializing":

- runSqliteMigration adds "Finalizing database schema" and "Copying database to
  SQLite" phases, start()/complete() around each stage (schema phase completes
  even on resume). Typed against a minimal structural view of the tracker.
- localized phase names across all locales.

Verified on a real upgrade: both phases surface in the package's update state
during the migration, and the node comes up on SQLite.
@helix-nine

Copy link
Copy Markdown
Contributor Author

Added init-progress reporting for the migration (f6b8d61).

The init framework hands each setupOnInit handler a FullProgressTracker as its 3rd arg (auto-syncs to setInitProgress in the background). The migration now reports two named phases so the StartOS update UI shows what's happening instead of a bare "initializing":

progress.addPhase('Finalizing database schema')  // chain 1
progress.addPhase('Copying database to SQLite')   // chain 2

Each is start()ed when its stage begins and complete()d when done (the schema phase completes even on a resumed run). Phase names are localized across all locales. runSqliteMigration's param is typed against a minimal structural view of the tracker so it doesn't reach into the SDK's non-exported class.

Verified on the same real upgrade path: during the conversion the package sits in updating and both phases show up in its progress state —

state=updating  phases_seen=['Finalizing database schema', 'Copying database to SQLite']

— then it flips to installed and the node comes up on SQLite (channel.sqlite/lnd.sqlite present, old bolt DBs tombstoned, lncli getinfo → 0.21.1-beta). So the migration still works end-to-end, now with visible progress.

MattDHill and others added 3 commits July 13, 2026 13:05
…rtos pins

Bumps @start9labs/start-sdk 2.0.3 -> 2.0.5 (dependency and lockfile only).

2.0.5 fixes ExVer range operations ignoring the downstream revision:
compareVersionRangePoints and adjacentVersionRangePoints compared the upstream
twice, so two points sharing an upstream but differing downstream (1.0.0:3 vs
1.0.0:15) collapsed into one. normalize() then dropped the lower of the two,
which made packed manifests advertise a canMigrateFrom/canMigrateTo range
narrower than the truth for any package whose `other` versions share current's
upstream. No upgrade path actually broke — StartOS resolves migrations through
the version graph rather than this field — but the manifests were wrong.

Also re-resolves the git-ref dependencies (bitcoin-core-startos, tor-startos) to the current head of the
branch each spec points at, so they carry start-sdk 2.0.5 too and dedupe
against the top-level copy instead of nesting a stale one.

No packaging API changed, so no source changes were needed. Verified with
`npm run check` (tsc --noEmit).
…mbeds the lnd version

UPDATING.md pointed at a dockerTag in startos/manifest/index.ts that does not
exist — the manifest uses dockerBuild, and the real pin is
`FROM lightninglabs/lnd:<ver>` in the Dockerfile.

It also missed a second pin entirely: `COPY --from=lightninglabs/lndinit:<tag>`,
whose tag embeds the lnd version (v0.1.36-beta-lnd-v0.21.1-beta). Bumping lnd
alone would point at a nonexistent lndinit image. Both pins are now documented,
each with an existence check.
…gration

Older LND (<v0.14.0-beta) created wtclient.db even when the watchtower
client was never used, leaving it at schema version 0. lndinit's
migrate-db version-checks channel.db and wtclient.db before copying and
hard-aborts if either isn't at its latest schema, so those legacy nodes
fail the bolt→SQLite conversion with 'refusing to migrate source
database with version 0'.

Activate the watchtower client in the schema-finalize run when a
wtclient.db is present, so LND applies its pending wtclient migrations
(in BuildDatabase, before unlock — the existing UNLOCKED gate already
covers it). Lossless: an empty (v0) db is just initialized; one from
prior watchtower use keeps its session data. Absent wtclient.db is left
untouched (lndinit skips a missing optional db).
@helix-nine

Copy link
Copy Markdown
Contributor Author

wtclient.db version-0 migration blocker

Added handling for a bolt→SQLite conversion failure @DRBoneZ hit on upgrade:

LNDINIT Runtime error: refusing to migrate source database with version 0 while
latest known DB version is 8; please upgrade the DB before using the data
migration tool or delete the wtclient.db manually ...

Cause. lndinit migrate-db version-checks exactly two DBs before copying — channel.db and wtclient.db — and hard-aborts the whole run if either isn't at its latest schema (cmd_migrate_db.gocheckWTClientDBMigrationsApplied). LND < v0.14.0-beta (commit 75531455daf0, Dec 2021) created wtclient.db unconditionally, but only wrote its schema version when the client was actually run — so any node whose data dir predates that (or briefly toggled the watchtower client on then off) carries an empty, version-0 wtclient.db. Current LND gates both on wtclient.active, so it can't produce this state; it's purely a legacy-data artifact. That's a large share of long-lived nodes, not an edge case.

Fix. In finalizeBoltSchema, when a wtclient.db is present, add --wtclient.active to the schema-finalize LND run. LND then applies the pending wtclient migrations during BuildDatabase (which completes before the wallet reaches UNLOCKED, so the existing gate already covers it), bringing the db to version 8 before lndinit runs.

Why activate-and-migrate rather than delete (the error's other suggestion): it's lossless in every case. An empty (v0) db is simply initialized to v8; a db from prior watchtower use (versions 1–7 are real, data-bearing schemas) keeps its session data. A blind rm would be safe only for the v0 case. An absent wtclient.db is left untouched — lndinit cleanly skips a missing optional db.

Verified against lndinit v0.1.36-beta and lnd v0.21.1-beta source (with an adversarial re-audit of the losslessness invariant and the db-discovery/skip logic).

Build: tsc --noEmit and the ncc bundle pass. Could not run the full s9pk pack locally (the shared buildx builder has a broken DNS resolver in this environment — unrelated to this diff, which touches no Dockerfile/manifest); CI runs the full pack.

Side note (out of scope, possible follow-up): the neutrino-based finalize run creates a fresh neutrino.db on bitcoind nodes that lndinit then copies to an inert neutrino.sqlite — harmless, but wastes some space and raises lndinit's peak free-space requirement. Worth a look separately.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants