Skip to content

p2p/sentry, node/eth: EIP-8159 eth/71 BAL fetcher + background downloader (PR 3/3)#20795

Merged
mh0lt merged 11 commits into
mainfrom
feat/eip-8159/fetcher-downloader
May 4, 2026
Merged

p2p/sentry, node/eth: EIP-8159 eth/71 BAL fetcher + background downloader (PR 3/3)#20795
mh0lt merged 11 commits into
mainfrom
feat/eip-8159/fetcher-downloader

Conversation

@mh0lt

@mh0lt mh0lt commented Apr 24, 2026

Copy link
Copy Markdown
Contributor

Summary

Third of three stacked PRs implementing EIP-8159. Adds the consumer-side BALFetcher primitive, a background BALDownloader that wires the fetcher into the node start path, and enables eth/71 in the default protocol-version list. Also ships a companion devnet-launch skill so bal-devnet-3 reproducers are reusable.

Depends on #20794 — review/merge that one first.

What lands

  • BALFetcher in p2p/sentry/sentry_multi_client/bal_fetcher.go:
    • Hash-mismatch → Sentry.PenalizePeer(Kick). Garbage or wrong-hash payloads never cross the p2p boundary.
    • Disambiguates empty-RLP (0xc0): accepted only when expected_hash == empty.BlockAccessListHash, treated as "peer has nothing" otherwise.
    • Writes validated BALs via rawdb.WriteBlockAccessListBytes.
  • BALDownloader in p2p/sentry/sentry_multi_client/bal_downloader.go: periodic scan for blocks whose header has a BlockAccessListHash and no local BAL entry; dispatches batches via the fetcher.
  • node/eth/backend.go spawns the downloader at startup. Always-on, negotiation-driven — silent no-op if no peer advertises eth/71.
  • node/nodecfg adds ETH71 to the default P2P protocol version list so erigon advertises eth/71 in its handshake.
  • .claude/skills/launch-bal-devnet-3/SKILL.md — companion skill for launching erigon + Lighthouse on the bal-devnet-3 ethpandaops devnet (parallels the existing launch-bal-devnet-2).

Deferred follow-ups

  • Silent-withholding peer scoring (the Phase 5b.2 plan item) is explicitly not in this PR. See the comment at bal_fetcher.go:89 — a per-peer score for repeated 0xc0 replies to non-empty expected hashes will build on top of this primitive. Will land as a follow-up once the substrate is reviewed.

Testing limitations

End-to-end verification on bal-devnet-3 is currently blocked by an unrelated execution bug at block 503 (EIP-8037 state-gas accounting, tracked in #20791). Reproduces deterministically on main with a fresh datadir and is not caused by anything in these PRs. When bal-devnet-4 goes live (currently no genesis/bootnodes yet), we'll re-run the end-to-end check there.

Local test coverage:

  • go test -short ./p2p/... (includes TestBlockAccessListsPacket66RoundTrip, TestAnswerGetBlockAccessListsQuery_OrderedResponseWithMissing, TestAnswerGetBlockAccessListsQuery_SoftSizeLimit, plus bal_fetcher_test.go)
  • make lint (0 issues, 2× determinism pass)
  • make erigon integration

Stack

  1. p2p/protocols/eth: EIP-8159 eth/71 wire protocol (PR 1/3) #20793 — wire protocol constants + packet types.
  2. p2p/protocols/eth, p2p/sentry: EIP-8159 eth/71 handler + sentry dispatch (PR 2/3) #20794 — handler + sentry dispatch.
  3. This PR — fetcher + background downloader + default protocol version + devnet-3 skill.

mh0lt and others added 6 commits April 28, 2026 05:59
…se 5a)

Subscribes to GET_BLOCK_ACCESS_LISTS_71 and routes it to a new handler
that answers with BlockAccessLists sourced from rawdb via the handler
added in Phase 3. After this commit, two erigon nodes running with the
eth/71-aware stack can complete the request/response round trip at the
wire level: node A sends GetBlockAccessLists; node B decodes, looks up
the BALs, and replies with BlockAccessLists positionally aligned.

Changes in p2p/sentry/sentry_multi_client/sentry_multi_client.go:

- RecvUploadMessageLoop subscribes to the new request MessageId
  (GET_BLOCK_ACCESS_LISTS_71) alongside the existing GetBlockBodies /
  GetReceipts subscriptions.

- New method getBlockAccessLists71 mirrors getBlockBodies66: decode the
  eth/66 request-id envelope, open a read-only tx, call the Phase 3
  handler (eth.AnswerGetBlockAccessListsQuery), encode the reply as
  BlockAccessListsPacket66 with the matching request id, and send via
  sentry.SendMessageById to BLOCK_ACCESS_LISTS_71.

- Placeholder blockAccessLists71 (no-op) is wired for inbound responses so
  the sentry routing table doesn't error. The full response path —
  request-id matching, keccak256 validation against the header's
  BlockAccessListHash, bad-peer scoring, and writing to rawdb — lives in
  the client fetcher landing next (Phase 5b).

- handleInboundMessage switch now routes both new MessageIds.

Tested: short tests pass in p2p/sentry, p2p/sentry/libsentry, and
p2p/sentry/sentry_multi_client; make lint clean; make erigon builds.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ase 5b.1)

Client-side fetcher for the eth/71 GetBlockAccessLists / BlockAccessLists
round trip, with header-hash validation and immediate bad-peer penalty on
mismatch. This is the primitive the next commit wires behind a debug RPC
method (for running a pair of bal-devnet-3 nodes against each other), and
on top of which the eventual stage-integrated fetcher will build.

New file: p2p/sentry/sentry_multi_client/bal_fetcher.go

- BALFetcher: request-id → waiting-goroutine map, mutex-protected.
- NewBALFetcher(): constructor, shared across sentries by MultiClient.
- FetchBlockAccessLists(ctx, sentry, peerID, blockHashes, expectedHashes):
  encodes a GetBlockAccessListsPacket66 with a random request id, sends it
  via Sentry.SendMessageById to the given peer, blocks until the matching
  BlockAccessLists response arrives (or ctx cancel / 30s default timeout),
  then validates each payload.
- Validation semantics:
  * empty payload (0xc0) + expected == empty.BlockAccessListHash → accepted
    as a genuinely empty BAL, returned as 0xc0.
  * empty payload + expected != empty hash → "peer does not have it",
    returned as nil so the caller can retry from another peer. NOT a
    bad-peer signal.
  * non-empty payload with keccak256(payload) != expected → peer sent
    garbage. Sentry.PenalizePeer(Kick) immediately, return ErrBadBALResponse.
  * length > requested → peer misbehaves, same Kick treatment.
- Deliver(peerID, packet): called from the inbound message handler. Matches
  by RequestId and requires the peer id to match the one we asked (dropping
  responses from impostors silently). Non-blocking send to a buffered(1)
  channel so duplicates or late arrivals never leak goroutines.

Wire-up in sentry_multi_client.go:

- MultiClient gains a balFetcher field, constructed by NewMultiClient.
- blockAccessLists71 inbound handler decodes the eth/66 envelope and calls
  cs.balFetcher.Deliver(peerID, &packet). Unknown / wrong-peer / stale
  arrivals are silently dropped (not bad-peer signals by themselves).
- MultiClient.FetchBlockAccessLists(ctx, peerID, blockHashes, expectedHashes)
  selects the first ready sentry and delegates to BALFetcher. Returns an
  error if no sentry is ready — matches the SendBodyRequest pattern.

Tests in p2p/sentry/sentry_multi_client/bal_fetcher_test.go — fake sentry
records SendMessageById / PenalizePeer calls and can deliver responses:

- ValidPopulatedResponse: populated BAL, keccak matches expected → accepted,
  no penalty.
- EmptyBALAcceptedOnlyWhenExpected: two-slot response where expected[0] is
  empty.BlockAccessListHash and expected[1] is some arbitrary hash; the
  peer returns 0xc0 for both. Slot 0 accepted as 0xc0, slot 1 returned as
  nil. No penalty in either case.
- HashMismatchPenalisesPeer: non-empty payload with wrong hash → returns
  ErrBadBALResponse and records a single PenaltyKind_Kick.
- DeliverIgnoresUnknownRequestID: Deliver with no matching in-flight entry
  returns false.
- DeliverIgnoresWrongPeer: impostor deliver from a different peer is
  rejected; target peer's later deliver succeeds.

Not in this commit:

- Withholding detection (peers that consistently return 0xc0 for BAL hashes
  known to be non-empty). That needs a rolling window and peer-score table
  which are natural to add alongside the stage integration in a later
  commit.
- A debug RPC method to expose FetchBlockAccessLists for devnet testing.
  Adding next (Phase 5b.2).
- Stage integration / background sync. Phase 5c.

Tested: make lint clean, make erigon builds, all 5 new tests pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Always-on background loop that fills rawdb with Block Access Lists for
recent blocks whose header commits to a BAL hash but whose BAL is not
yet stored locally. Rollout model matches eth/70: no feature flag,
behaviour is gated purely by per-peer capability negotiation. If no
connected peer advertises eth/71 every scan pass is a silent no-op;
once any peer negotiates eth/71 missing BALs start flowing in.

The block executor already regenerates and validates BALs locally via
ProcessBAL, so a missing p2p-delivered BAL is never a correctness issue
— only a CPU-cost optimisation. The downloader therefore runs strictly
in the background and never blocks stage progress; a failed or missing
fetch is retried on the next pass (default every 10s over a 256-block
window from head).

New file: p2p/sentry/sentry_multi_client/bal_downloader.go

- BALDownloader: holds MultiClient ref for peer picking + issuing
  GetBlockAccessLists, plus a writable kv.RwDB for persisting BALs
  (MultiClient's own db field is read-only by design).
- Run(ctx): 15s initial delay so sentries have time to negotiate, then
  a ticker loop that invokes scanAndFetch until ctx cancel.
- scanAndFetch: pickEth71Peer → collectMissingBALs → fetch in batches
  of 32 hashes with max-4 parallelism via a semaphore.
- collectMissingBALs: walks head..head-scanDepth, returns entries whose
  hdr.BlockAccessListHash is non-nil and whose BAL isn't in rawdb. Stops
  walking once it hits a pre-Amsterdam header.
- fetchBatch: calls MultiClient.FetchBlockAccessLists (which handles
  validation + bad-peer penalty via the BALFetcher from 5b.1). Accepted
  entries are written via rawdb.WriteBlockAccessListBytes. "Not available"
  slots (nil in response) are silently skipped for retry next pass.
- pickEth71Peer: iterates all sentries, calls Peers() RPC, filters by
  Caps containing "eth/71", picks one uniformly at random.

Wire-up in node/eth/backend.go:

- After sentriesClient is constructed, kick off `go
  NewBALDownloader(sentriesClient, chainDB, logger).Run(sentryCtx)`.
  Lifetime tied to sentryCtx so shutdown cancels the loop cleanly.

Tested: go build ./p2p/sentry/... ./node/eth/... clean; make lint 0
issues; make erigon builds. Existing sentry multi-client short tests
still pass.

This completes the Phase 5 stage-integration path. With the full stack
landed (Phases 1+2 wire protocol, 3 answer handler, 4 sentry dispatch,
5a server subscription, 5b.1 fetcher primitive, 5c downloader) two
erigon nodes running this branch on a BAL-enabled devnet will
negotiate eth/71 and exchange BALs end-to-end without any operator
intervention.

Next: Phase 6 — hive / integration tests and devnet verification
(bal-devnet-3 for immediate testing; bal-devnet-4 early next week).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Without this, eth/71 is registered in the message tables (Phase 1) and
the sentry dispatch (Phase 4) but the devp2p server never constructs a
listener for it — the default ProtocolVersion slice only includes
ETH69 and ETH70, so peers see capabilities "eth/69, eth/70" and never
negotiate eth/71 even when both sides support it.

Tested locally with two erigon instances (bal-devnet-3 chain, static
peers): before this patch the "Started P2P networking" log lines only
showed version=69 and version=70; with the patch eth/71 appears and
peers can exchange GetBlockAccessLists / BlockAccessLists.

Runtime override via --p2p.protocol-version still works for users who
want a narrower set.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a launch skill for the bal-devnet-3 ethpandaops devnet alongside the
existing launch-bal-devnet-2 skill. Structure is identical; diffs are the
devnet-specific constants (chain ID 7098917910, genesis/checkpoint-sync
URLs, Lighthouse image tag bal-devnet-3-65bb283, 15 vs 16 bootnodes, Dora
explorer URL).

Ships with the eth/71 (EIP-8159) PR so devnet-3 reproducers are reusable.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
)

Companion to the handler-side change. ethereum/EIPs#11553 made the
"not available" sentinel 0x80 (empty RLP string) — distinct from 0xc0
(empty RLP list = genuinely empty BAL). Old wire ambiguity is gone, so
the fetcher now does a clean three-way decode:

  0x80  → peer doesn't have it → out[i] = nil (caller retries elsewhere)
  0xc0  → peer claims empty BAL → accepted iff expected hash equals
          empty.BlockAccessListHash; otherwise hash-mismatch → kick
  else  → must hash to expected; otherwise kick

Pre-EIP-11553, returning 0xc0 with an expected non-empty hash was
silently treated as "unavailable" because we couldn't tell the peer
apart from one being honest about not having it. With 0x80 as the
explicit unavailable signal, that 0xc0-with-non-empty-hash case is
now unambiguous lying behaviour, so the fetcher kicks.

- bal_fetcher.go: three-way decode + comment refresh.
- bal_fetcher_test.go: existing empty-BAL test split into two —
  one for legitimate empty (0xc0+matching hash) plus 0x80 not-available;
  new test asserts 0xc0+non-empty-hash kicks the peer. Wrong-peer test
  uses 0x80 for the benign target reply.
- bal_downloader.go: comment cleanup; the 0xc0 special-case was a
  no-op (fetcher now hands the canonical empty-BAL bytes through, so
  the writer just persists whatever non-nil payload arrived).
- docs/plans/eip-8159-eth71-bal-exchange.md: update the empty-RLP
  ambiguity section to reflect the three-way decode.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@mh0lt mh0lt force-pushed the feat/eip-8159/fetcher-downloader branch from 3e7bb0e to 4bdf325 Compare April 28, 2026 06:00
@yperbasis yperbasis added Glamsterdam https://eips.ethereum.org/EIPS/eip-7773 Networking labels Apr 28, 2026

@yperbasis yperbasis left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

High-impact

  1. Multi-sentry routing silently breaks SendMessageById

pickEth71Peer records the sentry index where each candidate peer lives (sentryI) but
https://github.com/erigontech/erigon/blob/feat/eip-8159/fetcher-downloader/p2p/sentry/sentry_multi_client/bal_downloader.go#L297-L301:
type candidate struct {
peerID [64]byte
sentryI int // collected but never returned
}
...
return c.peerID, true, nil

MultiClient.FetchBlockAccessLists (sentry_multi_client.go:642) then picks an independent sentry via randSentryIndex() — the same pattern the body/header downloaders use, but those
send via SendMessageByMinBlock (broadcast). For SendMessageById, this is broken in multi-sentry mode: GrpcServer.SendMessageById returns (reply, nil) with empty reply.Peers when the
peer isn't on that sentry (sentry_grpc_server.go:1276-1280, including the explicit //TODO: enable after support peer to sentry mapping). The fetcher only checks err, so it registers
the in-flight request and waits the full 30 s defaultFetchTimeout. With N sentries, ~(N-1)/N requests time out before any progress.

In production today it's almost always a single sentry, so this is latent — but the sentryI collection and discard implies the author was aware and the wiring is incomplete. Either:

  • thread sentryI through MultiClient.FetchBlockAccessLists and use that exact sentry, or
  • route through sentryProvider.Multiplexer (which fans out across all sentries — node/components/sentry/provider.go:326).
  1. SendMessageById zero-peers reply not handled (bal_fetcher.go:167-169)

Even on the correct sentry, the peer can disconnect between pickEth71Peer and SendMessageById. The grpc server then returns (reply, nil) with reply.Peers == nil. Fetcher silently
waits 30 s. Worth adding:
sent, err := sentry.SendMessageById(ctx, &outreq, &grpc.EmptyCallOption{})
if err != nil { return nil, ... }
if sent == nil || len(sent.Peers) == 0 { return nil, ErrPeerGone /* or similar */ }

  1. Head-reorg → innocent peers kicked (bal_downloader.go:177-209, bal_fetcher.go:202-204)

collectMissingBALs snapshots header.BlockAccessListHash from the canonical chain at scan time. With scanInterval=10s and defaultFetchTimeout=30s, a head reorg can complete between
scan and validation. The fetcher then sees the BAL bytes for the new canonical block, computes a non-matching hash against the old expected hash, and PenalizePeer(Kick)s an honest
peer. Restricting reorgs to the recent head suggests this only affects the last few entries, but it's still a real false-positive kick path.

Two reasonable mitigations: (a) skip expected != latest_canonical_hash slots silently rather than kick, or (b) re-fetch the canonical header.BlockAccessListHash inside fetchBatch
immediately before the kick.

Medium

  1. Batch size cuts close to soft-limit (bal_downloader.go:121)

Constant batchSize = 32 × spec's "approximately 70 KiB" typical BAL ≈ 2.2 MiB, which exceeds the 2 MiB softResponseLimit enforced in AnswerGetBlockAccessListsQuery. Most batches will
hit the limit, peer truncates, fetcher pads with nil, downloader retries the trailing slots next pass. Functionally correct but wasteful. Drop to ~24 or compute against actual
observed BAL size on this chain.

  1. DB read errors swallowed (bal_downloader.go:196)
    existing, _ := rawdb.ReadBlockAccessListBytes(tx, hash, n)
    Any real DB error gets re-treated as "missing", refetched, written, refetched, … forever. Either propagate or at least Debug-log.

  2. Peers() errors silently dropped (bal_downloader.go:281-283)
    reply, err := sc.Peers(ctx, &emptypb.Empty{}, grpc.EmptyCallOption{})
    if err != nil { continue }
    Drops without logging — masks gRPC issues against external sentry.

  3. Test coverage gaps

  • bal_downloader.go (318 LOC) has zero unit tests. The PR description's go test -short ./p2p/... only covers the fetcher. collectMissingBALs walking, batching, parallel dispatch,
    persistence — all uncovered.
  • bal_fetcher_test.go is missing tests for: timeout (30 s deadline), ctx.Done(), more-entries-than-requested kick path, short response (M < N entries) → trailing nils, length mismatch
    error, and empty input early return.

Minor

  1. balRequest.deliver comment is wrong (bal_fetcher.go:69-70)

▎ "Closed on context cancel or timeout so a goroutine sending on it never leaks."

The channel is never closed. Leak-freedom comes from (a) buffered(1) + non-blocking send in Deliver, and (b) defer delete(f.inflight, reqID), which prevents future Delivers from
finding the request. That's fine but the comment misrepresents the mechanism.

  1. Hardcoded 15 s initial delay (bal_downloader.go:91) — not configurable; trivial for tests.

  2. Always-on across BAL-disabled chains (backend.go:711) — fine in practice because collectMissingBALs short-circuits on the first pre-Amsterdam header, but a
    chainConfig.AmsterdamTime != nil gate would be cheap to add and avoid even the first wake.

  3. getBlockAccessLists71 deferred-Rollback pattern (sentry_multi_client.go:670-674)

The explicit tx.Rollback() after AnswerGetBlockAccessListsQuery plus the defer tx.Rollback() mirrors getBlockBodies66 exactly. Per CLAUDE.md lint rules ("Never remove an explicit
.Close() or .Rollback() — add defer as a safety net alongside it"), this is intentional and consistent with the existing pattern.

Spec compliance ✅

  • Three-way decode (0x80 not-available, 0xc0 empty-BAL accepted only when expected == empty.BlockAccessListHash, else hash check) — implemented and tested.
  • Validation via keccak256(payload) == header.BlockAccessListHash — correct.
  • Positional alignment with shorter-than-requested allowed for soft-limit truncation — correct.
  • softResponseLimit = 2 MiB honored on server side; MaxBlockAccessListsServe = 1024.
  • Peer kick on hash mismatch — correct (per EIP §Security Considerations).
  • Silent-withholding scoring deferred — acknowledged in PR description; reasonable.

Bottom line

The fetcher's validation/kick logic is solid and the BALFetcher primitive is well-shaped. The downloader's plumbing has the multi-sentry routing bug (#1) which is the only thing I'd
consider blocking, and the reorg-kick (#3) and soft-limit-overshoot (#4) are real but not urgent. The test coverage gap around bal_downloader.go (#7) is the biggest follow-up — easy
to land in a separate PR but worth at least a smoke test for collectMissingBALs.

@yperbasis yperbasis linked an issue Apr 28, 2026 that may be closed by this pull request
Addresses the high/medium/minor items from yperbasis's review:

High-impact:
- Multi-sentry SendMessageById routing: pickEth71Peer now returns
  (peer, sentryIndex, ...); MultiClient.FetchBlockAccessLists takes
  the sentry index explicitly so the request reaches the same sentry
  the peer is connected via. Without this, ~(N-1)/N requests on N
  sentries timed out waiting for an empty Peers reply.
- Zero-peers reply: SendMessageById can succeed with empty Peers when
  the peer is unreachable on the chosen sentry or disconnected
  between selection and send. Added ErrPeerGone check so we return
  immediately instead of waiting the full defaultFetchTimeout.
- Head-reorg false-positive peer kicks: fetchBatch now re-validates
  each entry's expected BAL hash against the live canonical view
  just before sending the request, dropping reorged/pruned entries
  silently rather than letting the fetcher kick an honest peer for
  a hash mismatch caused by our own stale scan snapshot.

Medium:
- batchSize 32 -> 24 (32 * ~70 KiB ≈ 2.2 MiB exceeded the 2 MiB
  softResponseLimit; peers had to truncate every response).
- ReadBlockAccessListBytes errors are now logged at Debug and
  the slot is skipped, rather than treated as "missing" (which
  produced refetch-rewrite-refetch loops on persistent DB issues).
- Peers() RPC errors now logged at Debug per sentry instead of
  being silently dropped.
- Added 6 BALFetcher tests: ErrPeerGone, length mismatch, empty
  input, ctx.Cancel, timeout/leak-freedom, too-many-entries kick,
  short-response trailing-nil pad.
- Added bal_downloader_test.go covering peerSupportsEth71 and
  pickEth71PeerFromSentries (no-candidates, sentry-index recording,
  Peers-error skip, not-ready skip, bad-hex skip).

Minor:
- chainConfig.AmsterdamTime gate around BALDownloader startup.
- Corrected balRequest.deliver doc comment to describe the actual
  leak-freedom mechanism (buffered(1) + delete-from-inflight in the
  defer, not channel close).

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Implements the consumer side of EIP-8159 (eth/71 Block Access List exchange) by adding a BALFetcher primitive, a background BALDownloader that backfills missing BALs into rawdb, and enabling eth/71 advertisement by default. This integrates BAL fetching into node startup while keeping it negotiation-driven (no-op when no peers support eth/71).

Changes:

  • Add BALFetcher (request/response correlation + hash/sentinel validation + peer penalization on bad payloads).
  • Add BALDownloader background loop to scan recent canonical blocks for missing BALs and fetch/store them in batches.
  • Enable eth/71 in defaults and start the downloader at node startup (fork-gated on Amsterdam).

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
p2p/sentry/sentry_multi_client/sentry_multi_client.go Wires eth/71 message handling into MultiClient and exposes FetchBlockAccessLists.
p2p/sentry/sentry_multi_client/bal_fetcher.go New BAL fetcher with request-id routing, sentinel handling, and hash validation.
p2p/sentry/sentry_multi_client/bal_fetcher_test.go Unit tests covering validation, timeouts/cancelation, spoofing, and penalty behavior.
p2p/sentry/sentry_multi_client/bal_downloader.go New background downloader scanning for missing BALs and fetching/storing them.
p2p/sentry/sentry_multi_client/bal_downloader_test.go Tests for peer selection logic and eth/71 capability detection.
node/nodecfg/defaults.go Adds ETH71 to the default advertised protocol versions.
node/eth/backend.go Starts the BAL downloader at node startup (Amsterdam-gated).
docs/plans/eip-8159-eth71-bal-exchange.md Updates plan docs to reflect 0x80 “not available” sentinel and three-way decoding.
.claude/skills/launch-bal-devnet-3/SKILL.md Adds a devnet-3 launch skill for reproducible testing.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

if ready, ok := sc.(interface{ Ready() bool }); ok && !ready.Ready() {
continue
}
reply, err := sc.Peers(ctx, &emptypb.Empty{}, grpc.EmptyCallOption{})

Copilot AI Apr 28, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This call uses grpc.EmptyCallOption{} (value) whereas the surrounding codebase consistently passes &grpc.EmptyCallOption{} (pointer) for gRPC call options. For consistency (and to avoid any subtle interface-implementation differences across grpc-go versions), update this to match the existing pattern used elsewhere in sentry_multi_client.

Suggested change
reply, err := sc.Peers(ctx, &emptypb.Empty{}, grpc.EmptyCallOption{})
reply, err := sc.Peers(ctx, &emptypb.Empty{}, &grpc.EmptyCallOption{})

Copilot uses AI. Check for mistakes.
eth.ToProto[direct.ETH69][eth.GetReceiptsMsg],
eth.ToProto[direct.ETH70][eth.GetReceiptsMsg],
// eth/71 (EIP-8159) BAL exchange
eth.ToProto[direct.ETH71][eth.GetBlockAccessListsMsg],

Copilot AI Apr 28, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GetBlockAccessLists requests are now subscribed on the upload stream, but BlockAccessLists responses are not subscribed anywhere (e.g. RecvMessageLoop IDs). Without subscribing to eth.ToProto[direct.ETH71][eth.BlockAccessListsMsg], BLOCK_ACCESS_LISTS_71 messages will never reach handleInboundMessage, so BALFetcher.Deliver/FetchBlockAccessLists will time out and the downloader won't make progress.

Suggested change
eth.ToProto[direct.ETH71][eth.GetBlockAccessListsMsg],
eth.ToProto[direct.ETH71][eth.GetBlockAccessListsMsg],
eth.ToProto[direct.ETH71][eth.BlockAccessListsMsg],

Copilot uses AI. Check for mistakes.
var packet eth.BlockAccessListsPacket66
if err := rlp.DecodeBytes(inreq.Data, &packet); err != nil {
return fmt.Errorf("decoding blockAccessLists71: %w, data: %x", err, inreq.Data)
}

Copilot AI Apr 28, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

blockAccessLists71 assumes cs.balFetcher is non-nil and will panic if MultiClient is constructed without it (there are unit tests in this package that build &MultiClient{...} directly). Consider guarding with a nil-check (drop the message) or ensuring all test constructors initialize balFetcher to keep inbound message handling panic-free.

Suggested change
}
}
if cs.balFetcher == nil {
return nil
}

Copilot uses AI. Check for mistakes.

@yperbasis yperbasis left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚨 New finding — Critical (blocking)

BLOCK_ACCESS_LISTS_71 (the response side) is not subscribed in any Recv*Loop. The fetcher's wire path is end-to-end broken in production.

sentry_multi_client.go:93 adds eth.ToProto[direct.ETH71][eth.GetBlockAccessListsMsg] (request) to RecvUploadMessageLoop but BlockAccessListsMsg (response) is missing from
RecvMessageLoop (line 121-129).

Trace for an inbound BlockAccessLists:

  1. Peer's reply hits eth.BlockAccessListsMsg case at sentry_grpc_server.go:492.
  2. hasSubscribers(MessageId_BLOCK_ACCESS_LISTS_71) returns false (no addMessagesStream call ever registered it).
  3. → continue — message dropped silently. givePermit = true is also never reached, so the per-peer permit accounting may leak too.
  4. Fetcher's Deliver is never called via real wire traffic; every FetchBlockAccessLists returns ErrFetchTimeout after 30s.

Confirmed by exhaustive search: BLOCK_ACCESS_LISTS_71 appears in ToProto/FromProto/ProtoIds, the gRPC fanout, and handleInboundMessage — but in zero MessagesRequest{Ids: [...]} calls
anywhere in the tree.

Why this wasn't caught:

  • Unit tests use fakeSentry that calls Deliver directly — bypassing the gRPC subscription path entirely.
  • The PR description acknowledges end-to-end devnet verification is blocked by the unrelated EIP-8037 bug at block 503; the earlier "two-node round trip" claim must have validated
    capability negotiation only, not actual BAL exchange.

Fix: add eth.ToProto[direct.ETH71][eth.BlockAccessListsMsg] to RecvMessageLoop's ids (mirrors how BlockBodiesMsg is wired). One line, plus an integration-style test that exercises the
gRPC stream path (or a smoke test with SentryClientDirect) so this regression class can't recur silently.

Partially addressed: downloader test coverage

The new bal_downloader_test.go (191 LOC) covers peerSupportsEth71 and pickEth71PeerFromSentries thoroughly — appropriate, since multi-sentry routing was the highest-risk fix in this
revision. But collectMissingBALs, fetchBatch, scanAndFetch, and Run still have no tests. These are the largest pieces of new code:

  • collectMissingBALs walking semantics (scanDepth boundary, n==0 termination, pre-Amsterdam break, DB-error continue path)
  • fetchBatch reorg revalidation logic (the high-impact fix from this revision)
  • scanAndFetch parallel-batch dispatch and ctx cancellation under load

A smoke test against a MockBlockReader + in-memory kv.RwDB + fakeSentry covering at least the reorg-revalidation drop path and len(existing) > 0 skip would catch most regressions
cheaply. Reasonable to land separately, but the highest-impact behavior added in 763d39c (reorg revalidation) is currently asserted only by inspection.

Minor (non-blocking)

  1. TestBALFetcher_TimeoutReturnsErrFetchTimeout doesn't test the timer branch. It uses context.WithTimeout(100ms), so the select fires ctx.Done() and returns ctx.Err(), not
    ErrFetchTimeout. The test only asserts err != nil and inflight cleanup, so it passes — but the 30s time.NewTimer path at bal_fetcher.go:198-199 is uncovered. A configurable timeout
    (or defaultFetchTimeout as a struct field) would let the test exercise that branch.
  2. collectMissingBALs continue-then-underflow. When ReadBlockAccessListBytes errors at n == 0, we continue; the loop post-statement n-- underflows to MaxUint64; next iteration
    HeaderByNumber(MaxUint64) likely returns nil and breaks. One extraneous iteration on an exotic error path — wasteful, not buggy. Cheapest fix: move the if n == 0 { break } check to
    right before the n-- increment, or restructure as for n := headNum; ; n-- { ... if n == 0 { break } }.
  3. All 4 concurrent batches dispatch to the same peer. pickEth71Peer is called once at the top of scanAndFetch; the maxConcurrent: 4 workers all use that single peer/sentryI pair.
    Other Erigon downloaders rotate peers per batch. For background priority work this is fine, but a slow peer serializes all four slots. Consider per-batch peer rotation as a follow-up.
  4. keccak256 helper duplication at bal_fetcher.go:285-289 — minor, but crypto.Keccak256 likely exists at the standard location. Worth a follow-up grep but not worth a churn cycle
    here.

Bottom line

The fetcher's validation/kick logic, fetch-batch reorg-aware revalidation, and multi-sentry routing fix are all solid. The blocker is the missing BlockAccessListsMsg subscription —
without it, no real BAL response ever reaches the fetcher, so the whole stack is a no-op against actual peers. One-line wire-up fix + a test that exercises the gRPC stream end-to-end.
Test coverage for the downloader's loop bodies remains the biggest follow-up but is reasonable to land separately.

Three follow-ups from yperbasis on PR #20795:

1. bal_downloader.go: pass &grpc.EmptyCallOption{} (pointer) instead of
   the value form to sc.Peers, matching every other call site in
   sentry_multi_client (bal_fetcher.go, broadcast.go, sentry_api.go).
   Defensive against grpc-go versions where the interface assertion
   may differ between value and pointer receivers.
   Ref: #20795 (comment)

2. sentry_multi_client.go RecvMessageLoop: subscribe to
   eth.ToProto[direct.ETH71][eth.BlockAccessListsMsg]. Without this,
   peer responses to outbound GetBlockAccessLists requests never reach
   HandleInboundMessage / blockAccessLists71, and every BAL fetch
   times out at the fetcher's pending map. Confirmed empirically on
   bal-devnet-3 — node-b was logging "[bal-downloader] fetch failed
   err=bal: fetch timed out waiting for peer response" continuously
   pre-fix; zero failures post-fix.
   Ref: #20795 (comment)

3. sentry_multi_client.go blockAccessLists71: nil-guard cs.balFetcher.
   In-package unit-test builds construct &MultiClient{...} directly
   without setting the BAL fetcher; an unsolicited BLOCK_ACCESS_LISTS_71
   message would otherwise panic. Drop the message instead.
   Ref: #20795 (comment)
@mh0lt

mh0lt commented Apr 30, 2026

Copy link
Copy Markdown
Contributor Author

Addressed all three review threads in 760837d. Per-thread response below; commit message has the full context.

  1. grpc.EmptyCallOption{} value vs pointer (thread) — switched to &grpc.EmptyCallOption{} so it matches the rest of sentry_multi_client (bal_fetcher.go, broadcast.go, sentry_api.go).

  2. Missing BlockAccessListsMsg subscription in RecvMessageLoop (thread) — added the eth/71 inbound subscription. This was a real correctness bug, not a style nit. I caught it on bal-devnet-3 with two of our nodes running pre-fix: node-b was emitting [bal-downloader] fetch failed err="bal: fetch timed out waiting for peer response" on every batch (peers replied, but responses never reached blockAccessLists71 because no consumer was registered for the inbound message id). After deploying the fix to both nodes: zero fetch failures, BAL fetcher delivers normally. Thanks for catching it pre-merge.

  3. blockAccessLists71 nil-guard for cs.balFetcher (thread) — added the early-return.

Build clean, make lint clean, go test -short ./p2p/sentry/sentry_multi_client/... ./p2p/protocols/eth/... passes locally. Ready for re-review.

… fixes

Two tests guarding the round-2 review fixes from PR #20795:

* TestBlockAccessLists71_NilFetcherDropsMessage exercises the new
  nil-guard. Constructs &MultiClient{} without a balFetcher (the in-
  package test pattern that previously panicked), encodes a valid
  BlockAccessListsPacket66, and asserts both the direct
  blockAccessLists71 call and the HandleInboundMessage(BLOCK_ACCESS_
  LISTS_71) dispatch path drop the message cleanly with nil error.

* TestRecvMessageLoop_SubscribesToBlockAccessListsMsg is a structural
  guard against regressing the inbound subscription back out. The
  subscription itself lives inside RecvMessageLoop's hardcoded id
  list — extracting it just for testability would be an unhelpful
  refactor — so we assert the wiring contract that subscription
  depends on (eth.ToProto[ETH71][BlockAccessListsMsg] mapping to
  MessageId_BLOCK_ACCESS_LISTS_71). The behavioural proof that the
  subscription itself fires lives both in the
  HandleInboundMessage path of the nil-fetcher test above and in the
  bal-devnet-3 live-peer trace (zero "[bal-downloader] fetch failed"
  entries post-fix vs continuous timeouts pre-fix).

Round-trip RLP coverage for GetBlockAccessListsPacket66 /
BlockAccessListsPacket66 was already in protocol_test.go
(TestBlockAccessListsPacket66RoundTrip + TestEth71ProtocolRegistration).
AnswerGetBlockAccessListsQuery was already in handlers_test.go
(TestAnswerGetBlockAccessListsQuery_OrderedResponseWithMissing +
_SoftSizeLimit). No additions there were needed.
@mh0lt

mh0lt commented Apr 30, 2026

Copy link
Copy Markdown
Contributor Author

Added test coverage for the two correctness-relevant review fixes (65f1692):

  • TestBlockAccessLists71_NilFetcherDropsMessage — covers the nil-guard. Constructs &MultiClient{} without a balFetcher and asserts both direct blockAccessLists71 and dispatch via HandleInboundMessage(BLOCK_ACCESS_LISTS_71) drop the message cleanly with nil error rather than panicking.
  • TestRecvMessageLoop_SubscribesToBlockAccessListsMsg — structural guard against regressing the inbound subscription back out. Asserts eth.ToProto[ETH71][BlockAccessListsMsg] resolves to MessageId_BLOCK_ACCESS_LISTS_71 (the contract RecvMessageLoop depends on). Behavioural proof of the subscription firing is in the HandleInboundMessage path of the nil-fetcher test plus the bal-devnet-3 live-peer trace (zero [bal-downloader] fetch failed entries post-fix vs continuous timeouts pre-fix).

Round-trip RLP coverage for the eth/71 packet types was already in protocol_test.go (TestBlockAccessListsPacket66RoundTrip, TestEth71ProtocolRegistration); handler coverage was already in handlers_test.go (TestAnswerGetBlockAccessListsQuery_OrderedResponseWithMissing, _SoftSizeLimit) — no additions needed there.

@yperbasis yperbasis left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking issues (worth addressing in this PR or follow-ups)

  1. TestRecvMessageLoop_SubscribesToBlockAccessListsMsg doesn't actually test the subscription — sentry_multi_client_test.go:314-322 only asserts the
    eth.ToProto[ETH71][BlockAccessListsMsg] map. A regression that drops the entry from RecvMessageLoop's ids slice (sentry_multi_client.go:123-133) would NOT be caught — the very class
    of bug this guard is supposed to prevent. Cheap fix: extract the ids slice to a package-level var recvMessageIds = []sentryproto.MessageId{...} referenced by RecvMessageLoop and have
    the test assert membership. Avoids the "we extracted it just for testability" concern by having the loop genuinely read the var.
  2. collectMissingBALs underflow path is fragile — bal_downloader.go:188-222. The if n == 0 { break } lives inside the missing-append branch (line 219), so the continues at lines 209
    and 212 skip it. At n == 0 the post-step n-- wraps to MaxUint64, then the next iteration only exits because HeaderByNumber(MaxUint64) returns nil and hits the if hdr == nil { break }.
    Currently safe-by-luck; restructuring as for n := headNum; ; n-- { ... if n == low { break } } removes the silent reliance on a downstream nil return.
  3. bal_fetcher.go:236-238 recomputes keccak256(entry) twice on the kick path — once for bytes.Equal, once for the error formatter. Cache it locally:
    got := keccak256(entry)
    if !bytes.Equal(got, expected[:]) { ... ErrBadBALResponse, i, expected, got) }
  4. Stale doc comment in parent-PR territory — p2p/protocols/eth/protocol.go:413-418 still says "An empty RLP list (0xc0) in slot i has two possible meanings". Post-#11553 the wire is
    unambiguous (0x80 = unavailable, 0xc0 = genuinely empty). The plan doc was updated (docs/plans/eip-8159-eth71-bal-exchange.md) but this docstring on BlockAccessListsPacket was missed.
    Easy one-line fixup here or in a follow-up.
  5. bal_fetcher.go:285-289 reimplements keccak256 — common/crypto.Keccak256 already exists with a pooled state. Replace the local helper. (Yperbasis flagged in round 2; restating here
    as a follow-up.)
  6. Downloader test coverage gap — bal_downloader_test.go only covers peerSupportsEth71 / pickEth71PeerFromSentries. The most behavior-sensitive change in this revision — fetchBatch's
    reorg revalidation drop path (bal_downloader.go:240-264) — is currently asserted only by inspection. A smoke test against MockBlockReader + in-memory kv.RwDB + fakeSentry covering at
    least the reorg-drop and len(existing) > 0 skip paths would catch most regressions cheaply. Reasonable as a follow-up; worth not letting it slide far.

Observations (informational, no action requested)

  1. Use-after-rollback pattern in getBlockAccessLists71 — sentry_multi_client.go:683-687 calls tx.Rollback() then rlp.EncodeToBytes(response). response is []rlp.RawValue whose elements
    come from db.GetOne via rawdb.ReadBlockAccessListBytes, and kv_interface.go:360 documents "GetOne references a readonly section of memory that must not be accessed after txn has
    terminated". This exact pattern lives in getBlockBodies66 and is what CLAUDE.md guides toward (preserve explicit .Rollback() alongside defer), so it's a pre-existing repo convention
    rather than a new bug — likely safe in practice for short read txns because MDBX doesn't reclaim read pages until a writer commits a snapshot. Not a blocker for this PR; flagging the
    contract gap so the team can decide whether to revisit the convention separately.
  2. All 4 concurrent batches share one peer — pickEth71Peer runs once per scanAndFetch (bal_downloader.go:116); a slow peer serializes all four slots. Fine for background priority
    work; per-batch peer rotation is a reasonable follow-up but no urgency.
  3. No metrics — the plan doc mentions per-case fetcher metrics for production observability (docs/plans/eip-8159-eth71-bal-exchange.md:347); not in this PR. Reasonable to defer until
    the silent-withholding scoring layer arrives, since metrics shape will depend on it.
  4. TestBALFetcher_TimeoutReturnsErrFetchTimeout documents that it doesn't actually exercise the time.NewTimer branch (uses ctx-cancel instead). Making defaultFetchTimeout a struct
    field would let a future test cover the timer path in <1s. Minor.

mh0lt added 2 commits April 30, 2026 15:22
TestGetBlockAccessLists71_AnswersAndSends exercises the full inbound
GET → answer → send response cycle that pairs with the BAL inbound
subscription fix on the client side. Previously untested:

- decoding GetBlockAccessListsPacket66 from the wire
- looking up stored BALs via AnswerGetBlockAccessListsQuery against a
  real temporal MDBX
- encoding a positionally-aligned BlockAccessListsPacket66 with the
  matching RequestId
- sending it back via SendMessageById with id BLOCK_ACCESS_LISTS_71

Includes a balHeaderNumberReader stub satisfying the
services.FullBlockReader contract minimally — only HeaderNumber is
overridden, the rest panic on call to flag accidental coupling.

Coverage table for eth/71 / EIP-8159 wire exchange after this commit:

  | path                              | covered by                                      |
  |-----------------------------------|-------------------------------------------------|
  | RLP round-trip (request+response) | TestBlockAccessListsPacket66RoundTrip           |
  | wire/sentry id wiring             | TestEth71ProtocolRegistration                   |
  | server: AnswerGet…Query helper    | TestAnswerGetBlockAccessListsQuery_*            |
  | server: getBlockAccessLists71     | TestGetBlockAccessLists71_AnswersAndSends (new) |
  | client: blockAccessLists71 nil    | TestBlockAccessLists71_NilFetcherDropsMessage   |
  | client: inbound subscription      | TestRecvMessageLoop_SubscribesToBlockAccessLi…  |
  | fetcher (Deliver, sentinels, …)   | TestBALFetcher_* (13 tests)                     |
  | downloader peer selection         | TestPickEth71PeerFromSentries_* (6 tests)       |
@mh0lt mh0lt added this pull request to the merge queue May 4, 2026
Merged via the queue into main with commit 2d73b3d May 4, 2026
37 checks passed
@mh0lt mh0lt deleted the feat/eip-8159/fetcher-downloader branch May 4, 2026 22:58
github-merge-queue Bot pushed a commit that referenced this pull request May 18, 2026
…el split (#21250)

## Summary

- EIP-8159
([ethereum/EIPs#11553](ethereum/EIPs#11553),
merged 2026-04-21) split the BAL response "not available" and "genuinely
empty BAL" sentinels onto distinct wire values: `0x80` (RLP empty
string) for unavailable, `0xc0` (RLP empty list) for an empty BAL. They
no longer need header-hash disambiguation.
- The server handler (`AnswerGetBlockAccessListsQuery`) and the BAL
fetcher (`bal_fetcher.go`) were already updated to that semantics in
#20794 / #20795, but the `BlockAccessListsPacket` docstring in
`protocol.go` and the round-trip test in `protocol_test.go` still
described the old ambiguous-0xc0 behaviour.
- This PR aligns those comments and test-case labels with the current
spec. No behavioural change — comments only.

## Test plan

- [x] `make lint` clean
- [x] Existing `TestBlockAccessListsPacket66RoundTrip` still passes
(encoding/decoding paths untouched)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Glamsterdam https://eips.ethereum.org/EIPS/eip-7773 Networking

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement EIP-8159: eth/71 - Block Access List Exchange

3 participants