rpc, node: fix nil-pointer panic in gzip batch flush race#22338
Conversation
A JSON-RPC batch containing 2+ streamable methods ran each call on its own goroutine, and each one invoked the shared gzip-streaming flush hook installed on the request context. gzipResponseWriter.Flush is not safe for concurrent use, so calling it from multiple goroutines raced on the underlying gzip.Writer and could dereference a nil flate compressor, crashing the node. Batch sub-calls write into private per-item buffers rather than the HTTP response writer, so the hook was never useful there in the first place. Mask it out of the context used by batch sub-call goroutines instead of hardening gzipResponseWriter itself, since it remains owned by a single goroutine everywhere else.
There was a problem hiding this comment.
Pull request overview
This PR fixes a concurrency bug in the JSON-RPC batch handling path where multiple batch sub-call goroutines could concurrently invoke the shared gzip “streaming flush” hook, racing on the underlying gzip.Writer and potentially crashing the node with a nil-pointer panic. The approach is to mask the gzip flush hook from the per-item batch goroutines (since they write to private buffers, not the HTTP response), and to add a regression test reproducing the prior race.
Changes:
- Add
withoutGzipStreamingHook(ctx)to explicitly mask any inherited gzip streaming flush hook for contexts used by batch sub-calls. - Apply this masking to the batch call processing context so concurrent batch items can’t call into the shared gzip response writer flush hook.
- Add a
-raceregression test that sends a real gzipped JSON-RPC batch of streamable calls through the gzip middleware and RPC server.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
rpc/http.go |
Adds a helper to mask inherited gzip streaming hook values in contexts used by batch sub-calls. |
rpc/handler.go |
Applies the hook-masking context wrapper for batch sub-call goroutines to prevent concurrent gzip flush usage. |
node/rpcstack_gzip_batch_race_test.go |
Adds a regression test intended to reproduce the prior concurrent flush race/panic under -race. |
Comments suppressed due to low confidence (1)
rpc/handler.go:210
- In handleBatch the waitgroup counter is initialized with len(msgs), but goroutines are started only for len(calls). If a batch contains any response-shaped message (msg.isResponse()==true), handleResponses won't add it to calls, so wg.Wait will deadlock and the request will hang (potential DoS). The waitgroup should track the number of spawned goroutines (len(calls)).
// Bounded parallelism pattern explanation https://blog.golang.org/pipelines#TOC_9.
boundedConcurrency := make(chan struct{}, h.maxBatchConcurrency)
defer close(boundedConcurrency)
wg := sync.WaitGroup{}
wg.Add(len(msgs))
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
yperbasis
left a comment
There was a problem hiding this comment.
The fix itself is correct and well-scoped. Two changes before merge:
1. Regression test asserts nothing (node/rpcstack_gzip_batch_race_test.go). It calls ServeHTTP and checks no result, so it passes even on an error / empty / uncompressed response — its only failure signal is -race (or a lucky panic). Make each call's payload exceed minGzipBodySize so the fixed path still gzips, then assert HTTP 200, Content-Encoding: gzip, and that the decoded batch contains all n responses. (This is also what Copilot's line-65 comment asks for; its line-29 "needs encoding/json" comment is only true once you add the unmarshalling.)
2. Pre-existing deadlock in handleBatch (rpc/handler.go:207) — not introduced here, but it's in the function you're editing. wg.Add(len(msgs)) while only len(calls) goroutines are spawned. handleResponses drops response-shaped messages and _subscription notifications from calls, so a batch mixing a call with either — e.g. [{"id":1,"method":"eth_chainId"},{"id":2,"result":"0x1"}] — leaves the counter non-zero, wg.Wait() hangs forever, and h.close's callWG.Wait() hangs with it, leaking two goroutines plus the connection per request (unauthenticated DoS; WaitGroup.Wait ignores ctx cancellation, so it survives client disconnect). Fix is wg.Add(len(calls)). Fine to split into a separate PR if you'd rather keep this one focused — but please land it with a repro test (TDD). This is the same issue Copilot flagged as "low confidence"; it's real.
Nit (optional): guard the hook call in runMethod with ok && flush != nil, so the mechanism doesn't depend on the value always being stored as an untyped nil.
… test Address review feedback on the gzip batch flush race fix: guard the runMethod flush call with a nil check so the mechanism doesn't rely on the hook always being stored as an untyped nil, and make WithGzipStreamingHook panic if ever called with a nil hook so misuse fails loudly at the call site instead of degrading silently. Strengthen TestGzipHandlerBatchConcurrentStreamableFlush to actually assert on the response (HTTP 200, gzip encoding, decoded batch contents) instead of only relying on -race to catch a regression.
|
@yperbasis Point 1 and point Nit are fixed in this PR. Point 2 will be addressed in separate PR |
yperbasis
left a comment
There was a problem hiding this comment.
Approving. Verified the root cause is the concurrent gzip.Writer race (not the empty-flush theory in the issue — in Go 1.25 an empty single-goroutine Flush is safe), and ran the tests under -race: the node test reliably reproduces the race once the masking line is removed, so it's a genuine regression guard.
Two optional nits, neither blocking:
- The three guard layers (mask the hook +
WithGzipStreamingHooknil-panic +runMethod's&& flush != nil) are somewhat redundant — the untyped-nil masking already makes a typed-nil hook unreachable. Fine to keep given the severity, but therunMethodguard adds little over the panic. withoutGzipStreamingHook's doc comment runs a touch long for our comment policy; the last sentence (untyped vs typed nil) could go.
Drop the last sentence explaining untyped-vs-typed nil; it restates what the code already shows.
…#22338) (#22383) ## Summary A JSON-RPC batch containing 2+ streamable methods ran each call on its own goroutine, and each one invoked the shared gzip-streaming flush hook installed on the request context. `gzipResponseWriter.Flush` is not safe for concurrent use, so calling it from multiple goroutines raced on the underlying `gzip.Writer` and could dereference a nil flate compressor, crashing the node. Batch sub-calls write into private per-item buffers rather than the HTTP response writer, so the hook was never useful there in the first place. Mask it out of the context used by batch sub-call goroutines instead of hardening `gzipResponseWriter` itself, since it remains owned by a single goroutine everywhere else. Also hardens the hook mechanism itself: `WithGzipStreamingHook` now panics if ever called with a nil hook, and `runMethod` guards the flush call with `ok && flush != nil`, so the mechanism doesn't rely on the hook always being stored as an untyped nil to stay safe. Fixes #22334 ## Tests - Added `node/rpcstack_gzip_batch_race_test.go` (`TestGzipHandlerBatchConcurrentStreamableFlush`): sends a real JSON-RPC batch of 8 calls to a streamable method through the actual gzip middleware and `rpc.Server`, with `Accept-Encoding: gzip`, and each call's payload sized above `minGzipBodySize` so the fixed path still gzips. Asserts `HTTP 200`, `Content-Encoding: gzip`, and that the decoded batch contains all `n` responses with the expected ids/results — not just reliance on `-race` to catch a regression. Before the fix, running it with `-race` reliably reproduced both the data race and, on some runs, the exact nil-pointer panic reported in the issue. After the fix it passes cleanly and repeatably under `-race` (verified over 15 consecutive runs). - Added `rpc/http_test.go` (`TestWithGzipStreamingHookPanicsOnNilHook`): pins that `WithGzipStreamingHook` panics when given a nil hook. - Added `rpc/handler_test.go` (`TestRunMethodFlushHookNilFunc`): pins that `runMethod` does not panic when the hook stored on the context is a typed nil `func()` rather than an untyped nil. - Full `rpc/...` and `node/...` suites pass with `-race`. ### Manual end-to-end verification Also reproduced the issue against two real running nodes, one built from this branch's parent commit and one from this branch, both sent the exact same JSON-RPC batch (16 calls to `debug_traceBlockByNumber`) with `Accept-Encoding: gzip`: - **Pre-fix binary**: crashed on the first request, with a panic identical to the one reported in the issue (`flate.(*Writer).Flush` → `gzip.(*Writer).Flush` → `gzipResponseWriter.Flush` → `runMethod` → `handleCall` → `handleCallMsg` → `handleBatch`). - **Fixed binary**: the same request succeeded (`HTTP 200`, correct JSON body) across 8 consecutive attempts, node stayed up and kept producing blocks throughout. ## Test plan - [x] `go test -race -run TestGzipHandlerBatchConcurrentStreamableFlush ./node/...` - [x] `go test -race -count=1 ./rpc/... ./node/...` - [x] `make lint` - [x] Manual repro against a real node: pre-fix binary crashes, fixed binary doesn't ## Note A pre-existing deadlock in `handleBatch` (`wg.Add(len(msgs))` vs. `len(calls)` goroutines spawned) was flagged during review. It's unrelated to the gzip race fixed here and will be addressed in a separate PR.
Summary
A JSON-RPC batch containing 2+ streamable methods ran each call on its own goroutine, and each one invoked the shared gzip-streaming flush hook installed on the request context.
gzipResponseWriter.Flushis not safe for concurrent use, so calling it from multiple goroutines raced on the underlyinggzip.Writerand could dereference a nil flate compressor, crashing the node.Batch sub-calls write into private per-item buffers rather than the HTTP response writer, so the hook was never useful there in the first place. Mask it out of the context used by batch sub-call goroutines instead of hardening
gzipResponseWriteritself, since it remains owned by a single goroutine everywhere else.Also hardens the hook mechanism itself:
WithGzipStreamingHooknow panics if ever called with a nil hook, andrunMethodguards the flush call withok && flush != nil, so the mechanism doesn't rely on the hook always being stored as an untyped nil to stay safe.Fixes #22334
Tests
node/rpcstack_gzip_batch_race_test.go(TestGzipHandlerBatchConcurrentStreamableFlush): sends a real JSON-RPC batch of 8 calls to a streamable method through the actual gzip middleware andrpc.Server, withAccept-Encoding: gzip, and each call's payload sized aboveminGzipBodySizeso the fixed path still gzips. AssertsHTTP 200,Content-Encoding: gzip, and that the decoded batch contains allnresponses with the expected ids/results — not just reliance on-raceto catch a regression. Before the fix, running it with-racereliably reproduced both the data race and, on some runs, the exact nil-pointer panic reported in the issue. After the fix it passes cleanly and repeatably under-race(verified over 15 consecutive runs).rpc/http_test.go(TestWithGzipStreamingHookPanicsOnNilHook): pins thatWithGzipStreamingHookpanics when given a nil hook.rpc/handler_test.go(TestRunMethodFlushHookNilFunc): pins thatrunMethoddoes not panic when the hook stored on the context is a typed nilfunc()rather than an untyped nil.rpc/...andnode/...suites pass with-race.Manual end-to-end verification
Also reproduced the issue against two real running nodes, one built from this branch's parent commit and one from this branch, both sent the exact same JSON-RPC batch (16 calls to
debug_traceBlockByNumber) withAccept-Encoding: gzip:flate.(*Writer).Flush→gzip.(*Writer).Flush→gzipResponseWriter.Flush→runMethod→handleCall→handleCallMsg→handleBatch).HTTP 200, correct JSON body) across 8 consecutive attempts, node stayed up and kept producing blocks throughout.Test plan
go test -race -run TestGzipHandlerBatchConcurrentStreamableFlush ./node/...go test -race -count=1 ./rpc/... ./node/...make lintNote
A pre-existing deadlock in
handleBatch(wg.Add(len(msgs))vs.len(calls)goroutines spawned) was flagged during review. It's unrelated to the gzip race fixed here and will be addressed in a separate PR.