Skip to content

rpc, node: fix nil-pointer panic in gzip batch flush race#22338

Merged
lupin012 merged 5 commits into
mainfrom
lupin012/fix_gzip_batch_flush_race
Jul 9, 2026
Merged

rpc, node: fix nil-pointer panic in gzip batch flush race#22338
lupin012 merged 5 commits into
mainfrom
lupin012/fix_gzip_batch_flush_race

Conversation

@lupin012

@lupin012 lupin012 commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Summary

A JSON-RPC batch containing 2+ streamable methods ran each call on its own goroutine, and each one invoked the shared gzip-streaming flush hook installed on the request context. gzipResponseWriter.Flush is not safe for concurrent use, so calling it from multiple goroutines raced on the underlying gzip.Writer and could dereference a nil flate compressor, crashing the node.

Batch sub-calls write into private per-item buffers rather than the HTTP response writer, so the hook was never useful there in the first place. Mask it out of the context used by batch sub-call goroutines instead of hardening gzipResponseWriter itself, since it remains owned by a single goroutine everywhere else.

Also hardens the hook mechanism itself: WithGzipStreamingHook now panics if ever called with a nil hook, and runMethod guards the flush call with ok && flush != nil, so the mechanism doesn't rely on the hook always being stored as an untyped nil to stay safe.

Fixes #22334

Tests

  • Added node/rpcstack_gzip_batch_race_test.go (TestGzipHandlerBatchConcurrentStreamableFlush): sends a real JSON-RPC batch of 8 calls to a streamable method through the actual gzip middleware and rpc.Server, with Accept-Encoding: gzip, and each call's payload sized above minGzipBodySize so the fixed path still gzips. Asserts HTTP 200, Content-Encoding: gzip, and that the decoded batch contains all n responses with the expected ids/results — not just reliance on -race to catch a regression. Before the fix, running it with -race reliably reproduced both the data race and, on some runs, the exact nil-pointer panic reported in the issue. After the fix it passes cleanly and repeatably under -race (verified over 15 consecutive runs).
  • Added rpc/http_test.go (TestWithGzipStreamingHookPanicsOnNilHook): pins that WithGzipStreamingHook panics when given a nil hook.
  • Added rpc/handler_test.go (TestRunMethodFlushHookNilFunc): pins that runMethod does not panic when the hook stored on the context is a typed nil func() rather than an untyped nil.
  • Full rpc/... and node/... suites pass with -race.

Manual end-to-end verification

Also reproduced the issue against two real running nodes, one built from this branch's parent commit and one from this branch, both sent the exact same JSON-RPC batch (16 calls to debug_traceBlockByNumber) with Accept-Encoding: gzip:

  • Pre-fix binary: crashed on the first request, with a panic identical to the one reported in the issue (flate.(*Writer).Flushgzip.(*Writer).FlushgzipResponseWriter.FlushrunMethodhandleCallhandleCallMsghandleBatch).
  • Fixed binary: the same request succeeded (HTTP 200, correct JSON body) across 8 consecutive attempts, node stayed up and kept producing blocks throughout.

Test plan

  • go test -race -run TestGzipHandlerBatchConcurrentStreamableFlush ./node/...
  • go test -race -count=1 ./rpc/... ./node/...
  • make lint
  • Manual repro against a real node: pre-fix binary crashes, fixed binary doesn't

Note

A pre-existing deadlock in handleBatch (wg.Add(len(msgs)) vs. len(calls) goroutines spawned) was flagged during review. It's unrelated to the gzip race fixed here and will be addressed in a separate PR.

lupin012 added 2 commits July 8, 2026 22:58
A JSON-RPC batch containing 2+ streamable methods ran each call on its
own goroutine, and each one invoked the shared gzip-streaming flush
hook installed on the request context. gzipResponseWriter.Flush is not
safe for concurrent use, so calling it from multiple goroutines raced
on the underlying gzip.Writer and could dereference a nil flate
compressor, crashing the node.

Batch sub-calls write into private per-item buffers rather than the
HTTP response writer, so the hook was never useful there in the first
place. Mask it out of the context used by batch sub-call goroutines
instead of hardening gzipResponseWriter itself, since it remains
owned by a single goroutine everywhere else.
@lupin012 lupin012 marked this pull request as ready for review July 9, 2026 05:13
@yperbasis yperbasis requested a review from Copilot July 9, 2026 08:33

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes a concurrency bug in the JSON-RPC batch handling path where multiple batch sub-call goroutines could concurrently invoke the shared gzip “streaming flush” hook, racing on the underlying gzip.Writer and potentially crashing the node with a nil-pointer panic. The approach is to mask the gzip flush hook from the per-item batch goroutines (since they write to private buffers, not the HTTP response), and to add a regression test reproducing the prior race.

Changes:

  • Add withoutGzipStreamingHook(ctx) to explicitly mask any inherited gzip streaming flush hook for contexts used by batch sub-calls.
  • Apply this masking to the batch call processing context so concurrent batch items can’t call into the shared gzip response writer flush hook.
  • Add a -race regression test that sends a real gzipped JSON-RPC batch of streamable calls through the gzip middleware and RPC server.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

File Description
rpc/http.go Adds a helper to mask inherited gzip streaming hook values in contexts used by batch sub-calls.
rpc/handler.go Applies the hook-masking context wrapper for batch sub-call goroutines to prevent concurrent gzip flush usage.
node/rpcstack_gzip_batch_race_test.go Adds a regression test intended to reproduce the prior concurrent flush race/panic under -race.
Comments suppressed due to low confidence (1)

rpc/handler.go:210

  • In handleBatch the waitgroup counter is initialized with len(msgs), but goroutines are started only for len(calls). If a batch contains any response-shaped message (msg.isResponse()==true), handleResponses won't add it to calls, so wg.Wait will deadlock and the request will hang (potential DoS). The waitgroup should track the number of spawned goroutines (len(calls)).
		// Bounded parallelism pattern explanation https://blog.golang.org/pipelines#TOC_9.
		boundedConcurrency := make(chan struct{}, h.maxBatchConcurrency)
		defer close(boundedConcurrency)
		wg := sync.WaitGroup{}
		wg.Add(len(msgs))

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread node/rpcstack_gzip_batch_race_test.go
Comment thread node/rpcstack_gzip_batch_race_test.go

@yperbasis yperbasis left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fix itself is correct and well-scoped. Two changes before merge:

1. Regression test asserts nothing (node/rpcstack_gzip_batch_race_test.go). It calls ServeHTTP and checks no result, so it passes even on an error / empty / uncompressed response — its only failure signal is -race (or a lucky panic). Make each call's payload exceed minGzipBodySize so the fixed path still gzips, then assert HTTP 200, Content-Encoding: gzip, and that the decoded batch contains all n responses. (This is also what Copilot's line-65 comment asks for; its line-29 "needs encoding/json" comment is only true once you add the unmarshalling.)

2. Pre-existing deadlock in handleBatch (rpc/handler.go:207) — not introduced here, but it's in the function you're editing. wg.Add(len(msgs)) while only len(calls) goroutines are spawned. handleResponses drops response-shaped messages and _subscription notifications from calls, so a batch mixing a call with either — e.g. [{"id":1,"method":"eth_chainId"},{"id":2,"result":"0x1"}] — leaves the counter non-zero, wg.Wait() hangs forever, and h.close's callWG.Wait() hangs with it, leaking two goroutines plus the connection per request (unauthenticated DoS; WaitGroup.Wait ignores ctx cancellation, so it survives client disconnect). Fix is wg.Add(len(calls)). Fine to split into a separate PR if you'd rather keep this one focused — but please land it with a repro test (TDD). This is the same issue Copilot flagged as "low confidence"; it's real.

Nit (optional): guard the hook call in runMethod with ok && flush != nil, so the mechanism doesn't depend on the value always being stored as an untyped nil.

lupin012 added 2 commits July 9, 2026 14:15
… test

Address review feedback on the gzip batch flush race fix: guard the
runMethod flush call with a nil check so the mechanism doesn't rely on
the hook always being stored as an untyped nil, and make
WithGzipStreamingHook panic if ever called with a nil hook so misuse
fails loudly at the call site instead of degrading silently.

Strengthen TestGzipHandlerBatchConcurrentStreamableFlush to actually
assert on the response (HTTP 200, gzip encoding, decoded batch
contents) instead of only relying on -race to catch a regression.
@lupin012

lupin012 commented Jul 9, 2026

Copy link
Copy Markdown
Contributor Author

@yperbasis Point 1 and point Nit are fixed in this PR. Point 2 will be addressed in separate PR

@yperbasis yperbasis left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. Verified the root cause is the concurrent gzip.Writer race (not the empty-flush theory in the issue — in Go 1.25 an empty single-goroutine Flush is safe), and ran the tests under -race: the node test reliably reproduces the race once the masking line is removed, so it's a genuine regression guard.

Two optional nits, neither blocking:

  • The three guard layers (mask the hook + WithGzipStreamingHook nil-panic + runMethod's && flush != nil) are somewhat redundant — the untyped-nil masking already makes a typed-nil hook unreachable. Fine to keep given the severity, but the runMethod guard adds little over the panic.
  • withoutGzipStreamingHook's doc comment runs a touch long for our comment policy; the last sentence (untyped vs typed nil) could go.

Drop the last sentence explaining untyped-vs-typed nil; it restates
what the code already shows.
@lupin012 lupin012 added this pull request to the merge queue Jul 9, 2026
Merged via the queue into main with commit 63cc7de Jul 9, 2026
92 checks passed
@lupin012 lupin012 deleted the lupin012/fix_gzip_batch_flush_race branch July 9, 2026 16:50
yperbasis pushed a commit that referenced this pull request Jul 13, 2026
…#22338) (#22383)

## Summary

A JSON-RPC batch containing 2+ streamable methods ran each call on its
own goroutine, and each one invoked the shared gzip-streaming flush hook
installed on the request context. `gzipResponseWriter.Flush` is not safe
for concurrent use, so calling it from multiple goroutines raced on the
underlying `gzip.Writer` and could dereference a nil flate compressor,
crashing the node.

Batch sub-calls write into private per-item buffers rather than the HTTP
response writer, so the hook was never useful there in the first place.
Mask it out of the context used by batch sub-call goroutines instead of
hardening `gzipResponseWriter` itself, since it remains owned by a
single goroutine everywhere else.

Also hardens the hook mechanism itself: `WithGzipStreamingHook` now
panics if ever called with a nil hook, and `runMethod` guards the flush
call with `ok && flush != nil`, so the mechanism doesn't rely on the
hook always being stored as an untyped nil to stay safe.

Fixes #22334

## Tests

- Added `node/rpcstack_gzip_batch_race_test.go`
(`TestGzipHandlerBatchConcurrentStreamableFlush`): sends a real JSON-RPC
batch of 8 calls to a streamable method through the actual gzip
middleware and `rpc.Server`, with `Accept-Encoding: gzip`, and each
call's payload sized above `minGzipBodySize` so the fixed path still
gzips. Asserts `HTTP 200`, `Content-Encoding: gzip`, and that the
decoded batch contains all `n` responses with the expected ids/results —
not just reliance on `-race` to catch a regression. Before the fix,
running it with `-race` reliably reproduced both the data race and, on
some runs, the exact nil-pointer panic reported in the issue. After the
fix it passes cleanly and repeatably under `-race` (verified over 15
consecutive runs).
- Added `rpc/http_test.go` (`TestWithGzipStreamingHookPanicsOnNilHook`):
pins that `WithGzipStreamingHook` panics when given a nil hook.
- Added `rpc/handler_test.go` (`TestRunMethodFlushHookNilFunc`): pins
that `runMethod` does not panic when the hook stored on the context is a
typed nil `func()` rather than an untyped nil.
- Full `rpc/...` and `node/...` suites pass with `-race`.

### Manual end-to-end verification

Also reproduced the issue against two real running nodes, one built from
this branch's parent commit and one from this branch, both sent the
exact same JSON-RPC batch (16 calls to `debug_traceBlockByNumber`) with
`Accept-Encoding: gzip`:

- **Pre-fix binary**: crashed on the first request, with a panic
identical to the one reported in the issue (`flate.(*Writer).Flush` →
`gzip.(*Writer).Flush` → `gzipResponseWriter.Flush` → `runMethod` →
`handleCall` → `handleCallMsg` → `handleBatch`).
- **Fixed binary**: the same request succeeded (`HTTP 200`, correct JSON
body) across 8 consecutive attempts, node stayed up and kept producing
blocks throughout.

## Test plan

- [x] `go test -race -run TestGzipHandlerBatchConcurrentStreamableFlush
./node/...`
- [x] `go test -race -count=1 ./rpc/... ./node/...`
- [x] `make lint`
- [x] Manual repro against a real node: pre-fix binary crashes, fixed
binary doesn't

## Note

A pre-existing deadlock in `handleBatch` (`wg.Add(len(msgs))` vs.
`len(calls)` goroutines spawned) was flagged during review. It's
unrelated to the gzip race fixed here and will be addressed in a
separate PR.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Panic (SIGSEGV / nil pointer) in gzipResponseWriter.Flush when serving a gzipped batch RPC request

3 participants