HTTPServer: Prevent race condition between worker thread and I/O thread by pinheadmz · Pull Request #35614 · bitcoin/bitcoin

pinheadmz · 2026-06-27T12:50:29Z

This prevents a losing race condition that could prevent the server from reading any more requests from an HTTP client.

Found and reported by the fuzzing department: dergoegge@7fe5f54

The Race:

A connected socket can either be written to or read from based on the result of GenerateWaitSockets(). That method checks the HTTPRemoteClient flag m_send_ready. If it's true the implication is that there is data in the client's send buffer ready to go. Once that data is sent and the buffer is empty, MaybeSendBytesFromBuffer() sets it false again.

The sad case was when a worker thread calling WriteReply() adds data to the send buffer, but before it sets m_send_ready to true, the I/O thread sends that data and empties the buffer. With the buffer unexpectedly empty, WriteReply() sets m_send_ready to true.

The effect of this is that the socket will stay in "write" mode with nothing to write. With nothing to write, MaybeSendBytesFromBuffer() never sets it back to false and the socket is stuck forever.

The Fix:

Simply move m_send_ready = true inside the block of WriteReply() where m_send_mutex is still held. This prevents the I/O thread from emptying the send buffer while the worker thread is setting the flag.

Testing:

To observe the race condition, revert the first commit "http: prevent race condition between worker thread and I/O thread" and run the unit test from the remainder of the branch. I like to see the logs:

`test_bitcoin --log_level=all --run_test=httpserver_tests -- --printtoconsole --debug=http --debug=lock'

The test will fail with a small probability. The socket will get stuck and the test will abort after a 60 second timeout. To garuntee the race condition loses and fail the test every time, slow down WriteReply() in the worker thread:

diff --git a/src/httpserver.cpp b/src/httpserver.cpp
index 99e30ff663..b0c7b516d8 100644
--- a/src/httpserver.cpp
+++ b/src/httpserver.cpp
@@ -614,6 +614,7 @@ void HTTPRequest::WriteReply(HTTPStatusCode status, std::span<const std::byte> r
     } else {
         // Inform HTTPServer I/O that data is ready to be sent to this client
         // in the next loop iteration.
+        std::this_thread::sleep_for(500ms);
         m_client->m_send_ready = true;
     }

With the first commit (the fix) back in place, slowing down the worker thread like this won't fail the test.

Bonus:

The unit test is spread over three commits. First, a method of the socket testing setup is templated so a mock socket that intentionally raises an error can be inserted. The unit test added in that commit covers a race condition that was fixed in #35182 in response to https://github.com/bitcoin/bitcoin/pull/35182/changes#r3358889539 so we get the added benefit of covering an error path, and guaranteeing coverage of both "optimistic send" (directly from worker thread) and regular send (from a tick in the I/O loop thread).

The next commit adds a worker thread to the unit test, at which point a race condition is possible but very unlikely because all requests are sent at once. Finally, we spread out the requests in the top commit and make the race condition much easier to catch.

DrahtBot · 2026-06-27T12:50:36Z

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/35614.

Reviews

See the guideline for information on the review process.

Type	Reviewers
ACK	janb84, dergoegge, theStack

If your review is incorrectly listed, please copy-paste  into the comment that the bot should ignore.

Conflicts

No conflicts as of last run.

DrahtBot · 2026-06-27T13:08:24Z

🚧 At least one of the CI tasks failed.
_{Task 32 bit ARM: https://github.com/bitcoin/bitcoin/actions/runs/28289717765/job/83819565264}
_{LLM reason (✨ experimental): CI failed during compilation because -Werror=overloaded-virtual treated a hidden virtual DynSock::operator=(Sock&&) in test/util/net.h (used by httpserver_tests.cpp) as an error.}

Hints

Try to run the tests locally, according to the documentation. However, a CI failure may still
happen due to a number of reasons, for example:

Possibly due to a silent merge conflict (the changes in this pull request being
incompatible with the current code in the target branch). If so, make sure to rebase on the latest
commit of the target branch.
A sanitizer issue, which can only be found by compiling with the sanitizer and running the
affected test.
An intermittent issue.

Leave a comment here, if you need help tracking down a confusing failure.

dergoegge

utACK f190c77

Fix looks correct to me.

janb84

Concept ACK f190c77

Code review; think it will resolve the ToCTOU but it uncovers a bit of a smell (IMHO) , see NIT.

janb84 · 2026-06-29T13:02:29Z

     * Set true by worker threads after writing a response to m_send_buffer.
     * Set false by the HTTPServer I/O thread after flushing m_send_buffer.
     * Checked in the HTTPServer I/O loop to avoid locking m_send_mutex if there's nothing to send.
+     * Must be set only while holding m_send_mutex.
     */
    std::atomic_bool m_send_ready{false};


Suggested change

* Set true by worker threads after writing a response to m_send_buffer.

* Set false by the HTTPServer I/O thread after flushing m_send_buffer.

* Checked in the HTTPServer I/O loop to avoid locking m_send_mutex if there's nothing to send.

* Must be set only while holding m_send_mutex.

*/

std::atomic_bool m_send_ready{false};

* Set true by worker threads after writing a response to m_send_buffer.

* Set false by the HTTPServer I/O thread after flushing m_send_buffer.

* Checked in the HTTPServer I/O loop to decide whether to poll the socket for

* writeability or readability.

* Guarded by m_send_mutex so it stays consistent with m_send_buffer's emptiness:

* the two must always be updated together under the same lock.

*/

bool m_send_ready GUARDED_BY(m_send_mutex){false};

NIT: I think the Comment uncovers a bit of a smell, guarded data wearing atomic's clothes. The invariant is m_send_ready ⇔ m_send_buffer emptiness, and both must move under the same lock. As far as I can see the field is only an atomic_bool to allow one lock-free read in GenerateWaitSockets and that loop already takes m_sock_mutex per client, the lock-free read saves essentially nothing. So imho no reason not to make it an bool m_send_ready GUARDED_BY(m_send_mutex){false};.

This turns the hand-maintained "must hold the lock" convention into a compile-time guarantee.

it needs also a change in httpserver.cpp, as mentioned:

diff --git a/src/httpserver.cpp b/src/httpserver.cpp index 92579ee106..35ed8c040f 100644 --- a/src/httpserver.cpp +++ b/src/httpserver.cpp @@ -941,7 +941,12 @@ HTTPServer::IOReadiness HTTPServer::GenerateWaitSockets() const // Check if client is ready to send data. Don't try to receive again // until the send buffer is cleared (all data sent to client). - Sock::Event event = (http_client->m_send_ready ? Sock::SEND : Sock::RECV); + // Keep this as a separate critical section from the m_sock_mutex one above: + // never hold m_sock_mutex and m_send_mutex at the same time here. + // MaybeSendBytesFromBuffer() locks m_send_mutex then m_sock_mutex, so nesting + // them in the opposite order here would risk a lock-order inversion deadlock. + const bool send_ready{WITH_LOCK(http_client->m_send_mutex, return http_client->m_send_ready;)}; + Sock::Event event = (send_ready ? Sock::SEND : Sock::RECV); io_readiness.events_per_sock.emplace(sock, Sock::Events{event}); io_readiness.httpclients_per_sock.emplace(sock, http_client); }

Thanks, I'll explicitly add the lock to the flag that makes sense.

This prevents a losing race condition that could prevent the server from reading requests from an HTTP client. A connected socket can either be written to or read from based on the result of GenerateWaitSockets(). That method checks the HTTPRemoteClient flag m_send_ready. If it's `true` the implication is that there is data in the client's send buffer ready to go. Once that data is sent and the buffer is empty, MaybeSendBytesFromBuffer() sets it `false` again. The sad case was when a worker thread calling WriteReply() adds data to the send buffer, but before it sets m_send_ready to `true`, the I/O thread sends that data and empties the buffer. With the buffer unexpectedly empty, WriteReply() sets m_send_ready to `true`. The effect of this is that the socket will stay in "write" mode with nothing to write. With nothing to write, MaybeSendBytesFromBuffer() never sets it back to `false` and the socket is stuck forever.

Implements a child class of DynSock which is used as the mock socket for HTTPServer unit tests. The ErrorSock::Send() method raises a non-permanent error on the first HTTPRequest::WriteReply() and then succeeds after the second. In httpserver_tests.cpp use this mechanism to ensure that the server retries a send operation if such an error is encountered, and cover both optimistic (worker thread WriteReply()) and non-optimistic (I/O thread SocketHandlerConnected()) send paths.

The result of WriteReply() losing the race condition would prevent any new requests being read from the socket. The socket error test sent 3 requests all at once after connecting, so in this commit we separate the the third request to make the losing race condition more likely, and make its effect more obvious.

pinheadmz · 2026-06-29T14:32:53Z

push to f595daf:

rebase on master, fix conflict with scripted-diff: Rename Sock::{RECV,SEND,ERR} #35588
Address review from @janb84, guarding the flag by the same lock as the buffer.

janb84

crACK f595daf

dergoegge

utACK f595daf

theStack

Code-review ACK f595daf

…05fc7ab 81405fc7ab Merge bitcoin/bitcoin#35689: test: Inline incorrect check in `util_tests` cd2a4bc510 test: Redeclare variable as signed in `util_tests` 6f1c56f03a Merge bitcoin/bitcoin#35670: net: optimize compact block extra tx iteration e94fda8a40 Merge bitcoin/bitcoin#35685: doc: Archive 30.3 release notes 5223cf1795 Merge bitcoin/bitcoin#35616: refactor: Use u64 over size_t for all cache sizes to fix a 32-bit overflow c0e91efdb3 Merge bitcoin/bitcoin#35295: validation: fetch block input prevouts in parallel during ConnectBlock f0da26cfc8 Merge bitcoin/bitcoin#34997: p2p: Don't participate in addr relay with feelers 443179a9eb doc: Archive 30.3 release notes fabafd91f1 refactor: Use u64 over size_t for all cache sizes to fix a 32-bit overflow 32941e1314 Merge bitcoin/bitcoin#35649: depends: move FreeBSD SDK handling to CI f379d716b1 Merge bitcoin/bitcoin#35669: doc: archive release notes for v31.1 881bb7b355 Merge bitcoin/bitcoin#35673: refactor: Move LoadGenesisBlock to ChainstateManager 3f3e644beb Merge bitcoin/bitcoin#35678: private broadcast: define and use new RPC_LIMIT_EXCEEDED error code ( + other follow-ups) e3b026bf56 Merge bitcoin/bitcoin#34020: mining: add getTransactions(ByWitnessID) IPC methods 62f9089343 Merge bitcoin/bitcoin#35386: doc: add an AI contribution policy fa615bd163 refactor: Move LoadGenesisBlock to ChainstateManager 8ac222484c private broadcast: remove no-op [[nodiscard]] 191bdcba26 test: align test better with described scenario 7ad311be18 test: use BOOST_CHECK_EQUAL for PrivateBroadcast::AddResult 82a02a2a22 rpc: define and use new RPC_LIMIT_EXCEEDED error code a64df338e6 Merge bitcoin/bitcoin#35651: doc: Improve offline-signing-tutorial after 32489 4498fa5d5b Merge bitcoin/bitcoin#35406: private broadcast: limit outstanding txs to count of 10,000 ec5edcd72b Merge bitcoin/bitcoin#35661: Update libmultiprocess subtree to add `ThreadMap.makePool` method ea67fea062 Merge bitcoin/bitcoin#35464: kernel: Add function for creating chainparams with a signet challenge a7b5f23bb8 Merge bitcoin/bitcoin#35667: refactor: Use `NetworkErrorString` for macOS code in `netif.cpp` 69fc991791 Merge bitcoin/bitcoin#32606: p2p: Drop unsolicited CMPCTBLOCK from non-HB peer and when blocksonly 1a3cbf1bd2 net: optimize compact block extra tx iteration bc33509ae2 Merge bitcoin/bitcoin#35650: doc: Add release notes for 32489 (exportwatchonlywallet RPC) 2b6e767d96 doc: archive release notes for v31.1 b0735336ee p2p: Don't participate in addr relay with feeler connections 302733fd96 Merge bitcoin/bitcoin#35652: init: fix reindex deadlock by waking cv after interrupt c1313b199f init: wake genesis wait after ImportBlocks() returns eccb04a321 refactor: Use `NetworkErrorString` for macOS code in `netif.cpp` f79ecfd9c7 Merge bitcoin/bitcoin#35658: refactor: Drop unneeded `<sys/types.h>` include before `<ifaddrs.h>` 22ac4ad949 ci: ensure we use correct lld version in OpenBSD job 495f43f7b3 ci: FreeBSD 15.1 244739db9d depends: move FreeBSD SDK handling to CI bab0120053 Merge bitcoin/bitcoin#35621: validation: Ignore eventual error message from flushing in AcceptBlock 02afa66169 Merge commit '6b0a907302364649dcaffeb0340b985f14141b4e' into pr/subtree-11 6b0a907302 Squashed 'src/ipc/libmultiprocess/' changes from 3edbe8f67c1..16bf05dea02 22c328d388 refactor: Drop unneeded `<sys/types.h>` include before `<ifaddrs.h>` 2063f02bd5 Merge bitcoin/bitcoin#35510: test: SOCKS5 proxy: expect that connection may be reset during SOCKS5 handshake or data forwarding 1360001f43 Merge bitcoin/bitcoin#34959: wallet: Enforce BDB btree levels and overflow item sizes 10ffef4b3b Merge bitcoin/bitcoin#35604: log: expose `-logratelimit` in normal help b393985aa0 Merge bitcoin/bitcoin#35634: txospenderindex: use zero-byte entry values 1835f2fcbf Merge bitcoin/bitcoin#35653: fuzz: Remove `ConsumeUniValue` 9f3e427228 fuzz: Remove ConsumeUniValue 68cb7840d2 doc: improve offline-signing-tutorial after 32489 32ddfc92d9 Merge bitcoin/bitcoin#35599: doc: Add release notes for #33671 (getbalances nonmempool field) f56804bcf5 Merge bitcoin/bitcoin#35640: ci: use a 8x instance over 16x for riscv job 239d6c5260 Merge bitcoin/bitcoin#35614: HTTPServer: Prevent race condition between worker thread and I/O thread cddbad325d doc: Add release notes for 32489 (exportwatchonlywallet RPC) 0a1bbec688 Merge bitcoin/bitcoin#32489: wallet: Add `exportwatchonlywallet` RPC to export a watchonly version of a wallet 4a007126fb Merge bitcoin/bitcoin#35147: depends: Boost 1.91.0-1 31abaa264c doc: add an AI contribution policy a15bdc0598 doc: update offline-signing-tutorial to use exportwatchonlywallet rpc a388076401 test: Test for exportwatchonlywallet d053e3e5c8 wallet, rpc: Add exportwatchonlywallet RPC 444878efef wallet: Add CWallet::ExportWatchOnly f9273f01db wallet: Move listdescriptors retrieving from RPC to CWallet a1c83789a7 wallet: Write new descriptor's cache in AddWalletDescriptor 1e996640e6 wallet: Use Descriptor::CanSelfExpand() in CanGetAddresses() d2ee9227da descriptor: Add CanSelfExpand() 2990bd7735 Merge bitcoin/bitcoin#35118: fuzz: add ipc round-trip fuzz target fb1d152c24 depends: Boost 1.91.0-1 ba48852f9e Merge bitcoin/bitcoin#35438: test: introduce NodeSigner, run feature_taproot.py without wallet compiled 47bbed052e ci: use true|false over "true|false" 9a25bc3989 ci: use a 8x instance over 16x 9871dc7ab4 Merge bitcoin/bitcoin#31425: CI: Add Riscv bare metal job 113402286e doc: add txospenderindex release note ce06878288 index: shrink txospenderindex value markers a99148d576 test kernel: Don't log on warnings change 5b4fd284f4 kernel: Generate a signet with a challenge 037ad77071 fuzz: add IPC round-trip target a8823c0996 Merge bitcoin/bitcoin#35607: nanobench: fix performance counter buffer initialization f26c15bdd2 Merge bitcoin/bitcoin#35397: ci: add OpenBSD Clang cross job 91586f701e test: introduce NodeSigner, run feature_taproot.py without wallet compiled 771200ca43 test: return full keypair from `getnewdestination` helper 4e29de719e private broadcast: add release note for limited cap cbf8c107c1 Release cs_main between individual private tx re-attempts 5aea3d0373 private broadcast: limit outstanding txs to count of 10,000 dc282ff31d Merge bitcoin/bitcoin#35597: logging: More fully remove libevent log category ec98037f7a Merge bitcoin/bitcoin#35615: fuzz: restore CreateSock in PCP targets 43d89bb6e1 Merge bitcoin/bitcoin#35610: bitcoin-util: Add netmagic command e35c805489 Merge bitcoin/bitcoin#35129: test: add fuzz test for private broadcast dc1c17c085 doc: add release notes 0e10937184 fuzz: add coins_view_stacked fuzz harness to test concurrent leveldb reads ce610a6ff4 fuzz: update harnesses to cover CoinsViewOverlay::StartFetching 760fb22dc3 test: add unit tests for CoinsViewOverlay::StartFetching d69a3b20de doc: update CoinsViewOverlay docstring to describe parallel fetching ab2a379237 coins: fetch inputs in parallel fdf283036a coins: add ready flag to InputToFetch ede11b8314 validation: collect block inputs in CoinsViewOverlay before ConnectBlock f82043af50 coins: introduce thread pool in CoinsViewOverlay 5bf1c32008 validation: add -prevoutfetchthreads configuration option a318f43254 bitcoin-util: Add netmagic command 256482ab56 validation: In AcceptBlock, ignore flush result 3c76bd4356 Merge bitcoin/bitcoin#35603: build: QRencode cleanups f595daf1dd test: ensure HTTPServer race condition is fixed b98b10c072 test: introduce a worker thread in http socket error test 922b08d375 test: socket error handling in HTTPServer using ErrorSock mock socket 73da2a8a52 http: prevent race condition between worker thread and I/O thread 57b3bf8496 Merge bitcoin/bitcoin#35609: ci: Bump tsan config to ubuntu:26.04 with -U_FORTIFY_SOURCE d64ea15824 ci: add openBSD cross CI job 5404b62074 depends: add openbsd_LDFLAGS 2a9e35d293 Merge bitcoin/bitcoin#35588: scripted-diff: Rename `Sock::{RECV,SEND,ERR}` a0e5e30010 fuzz: restore CreateSock in PCP targets 41ceea400e scripted-diff: Rename `StatusLevel::{INFO,WARN,ERR}` f395acdeee scripted-diff: Rename `Sock::{RECV,SEND,ERR}` 7a74f65293 Merge bitcoin/bitcoin#35536: fuzz: share a single mocked steady clock across FuzzedSock instances e1290ce7f7 Merge bitcoin/bitcoin#35543: test: introduce ExtendedPrivateKey and ExtendedPublicKey classes 7ac25c9177 util, refactor: Rename local `ERR` in `Sock::Accept` bbbbab86a8 ci: Bump tsan config to ubuntu:26.04 with -U_FORTIFY_SOURCE 095596ddf7 log: expose -logratelimit option ea9afb61a1 Merge bitcoin/bitcoin#35602: doc: Clarify build docs about `pkgconf` / `pkg-config` requirements 672eedc46b Merge bitcoin/bitcoin#35220: fuzz: connman: strengthen assertions and extend coverage 0e5c718d8a Merge bitcoin/bitcoin#35506: test: ensure group data cluster pointers are live b6b1d0653a nanobench: fix perf counter buffer init 93012d7ff9 Merge bitcoin/bitcoin#35601: wallet: remove experimental warning from send and sendall 829255c8be cmake: Remove `SelectLibraryConfigurations` from `FindQRencode` module 5c55606da9 depends: Remove unused `lib/pkgconfig` in `qrencode` package 402ba10b20 cmake: Drop optional `PkgConfig` use in `FindQRencode` module 8ebfff0f88 doc: add send RPC release note fb8a103886 doc: Clarify build docs about `pkgconf` / `pkg-config` requirements 5884f5a4fa wallet: remove experimental warning from send RPCs 7b84e5106c Merge bitcoin/bitcoin#35595: ci: remove some packages from Chimera job 58560c281d ci: remove some packages from Chimera job 295ce6f45c Merge bitcoin/bitcoin#35576: test: raise `feature_reindex` RPC timeout b36730a3ef Add CI job for riscv bare metal bfdbf513f6 Add CI job for producing a static bare metal binary a9a1d92a1d build: Add option for building for bare metal envs 9b2ef81757 doc: add release notes for #33671 (getbalances nonmempool field) 633044f143 Merge bitcoin/bitcoin#35266: rpc, wallet: add an option to not load the wallet after migrating d6269e2a90 Merge bitcoin/bitcoin#35594: fuzz: cover async chainstate compaction 3765b428d1 logging: More fully remove libevent log category 703a671fbc fuzz: compact coins view db during fuzzing 0868c85fd5 refactor: rename async coin compaction 6fa4132298 fuzz: share a single mocked steady clock across FuzzedSock instances 8791c4764c test: use ExtendedPrivateKey in wallet_taproot.py 89ceafafb9 test: use ExtendedPrivateKey in wallet_listdescriptors.py bbfffcab58 test: use ExtendedPrivateKey in wallet_send.py 2ab6e590f7 test: use ExtendedPrivateKey in wallet_keypool.py 9e20118720 test: use ExtendedPrivateKey in wallet_fundrawtransaction.py 06af0cddbb test: use ExtendedPrivateKey in wallet_descriptor.py 4100fac20e test: use ExtendedPrivateKey in wallet_createwallet.py ff3f6def9a test: use ExtendedPrivateKey in wallet_bumpfee.py 003f2a01f6 test: use ExtendedPrivateKey in feature_notifications.py f988e6d6e6 test: use ExtendedPrivateKey in wallet_importdescriptors.py 0cdd817a82 add release note 517d37ce3e test: tests wallet migration with load_wallet disabled b98dd63da7 rpc: Add load_wallet argument to migratewallet RPC 4acd063ba6 wallet: make loading the wallet after migrating optional 1a3cfdf1b7 fuzz: connman: cover AddLocalServices/RemoveLocalServices c507fb3063 fuzz: connman: add outbound-bytes invariants 4a6fce43ea fuzz: connman: add AddNode/RemoveAddedNode invariants 9e6546c517 test: raise reindex mining RPC timeout d2a03d50ac test: add extendedkey.py unit tests by using BIP32 test vectors afdb378082 test: introduce ExtendedPrivateKey and ExtendedPublicKey classes 55e3a57f22 qa: Avoid UTXO reuse between test functions 9c5dd2926a p2p: Ignore CMPCTBLOCK from peer that hasn't sent SENDCMPCT bf9884f4e5 p2p: make blocksonly nodes ignore CMPCTBLOCK messages 92cea63c71 test: (Un)solicited invalid cb -> get disconnected. e845e26344 test: p2p: Nodes ignore unsolicited CMPCTBLOCK's 8313591715 p2p: Drop unsolicited CMPCTBLOCK from non-HB peer 44f377a71f refactor: test: Static assert_highbandwidth_states 25457a3272 test: Tighten getblocktxn checks in parallel cb reconstruction test. 2ee4fafa3f test: add fuzz test for private broadcast 08b7c61fc7 private broadcast: enforce sending to unique node ids 9a8ef9b0a3 test: SOCKS5 proxy: expect that connection may be reset during handshake eb3208364a test: SOCKS5 proxy: expect that connection may be reset when forwarding 4dbaa7cc65 test: generalise byte_to_base58 utility function to allow more version types df9eb72b12 test: ensure group data cluster pointers are live a5859edef4 fuzz: connman: set m_local_services/m_use_addrman_outgoing/m_max_automatic_connections 4b84c9125a fuzz: connman: add network activity invariants 9784818442 mining: add getTransactionsByWitnessID() IPC method d282ae6883 mining: add getTransactionsByTxID() IPC method 0d5e4d4712 test: restart node after IPC option override test f16b3613cd ipc: Serialize null CTransactionRef as empty Data 0f466e1094 mempool: add lookup by witness hash 51dd90fb50 refactor: Merge announce_cmpct_block() defs into one 97d08d62ba refactor: store wallet names to MigrationResult 8a739a5510 build: allow ipc fuzz builds b2de59d486 wallet, bdbro: Validate btree page levels dc3a2b9c3b wallet, bdbro: Enforce overflow data lengths git-subtree-dir: libbitcoinkernel-sys/bitcoin git-subtree-split: 81405fc7abbd1889f3978b8924e7acbe12b3403b

…en worker thread and I/O thread cd95a5a test: ensure HTTPServer race condition is fixed (Matthew Zipkin) 429e29d test: introduce a worker thread in http socket error test (Matthew Zipkin) a6f7322 test: socket error handling in HTTPServer using ErrorSock mock socket (Matthew Zipkin) 7f2851e http: prevent race condition between worker thread and I/O thread (Matthew Zipkin) Pull request description: This prevents a losing race condition that could prevent the server from reading any more requests from an HTTP client. Found and reported by the fuzzing department: dergoegge/bitcoin@7fe5f54 The Race: A connected socket can either be written to or read from based on the result of `GenerateWaitSockets()`. That method checks the `HTTPRemoteClient` flag `m_send_ready`. If it's `true` the implication is that there is data in the client's send buffer ready to go. Once that data is sent and the buffer is empty, `MaybeSendBytesFromBuffer()` sets it `false` again. The sad case was when a worker thread calling `WriteReply()` adds data to the send buffer, but before it sets `m_send_ready` to `true`, the I/O thread sends that data and empties the buffer. With the buffer unexpectedly empty, `WriteReply()` sets `m_send_ready` to `true`. The effect of this is that the socket will stay in "write" mode with nothing to write. With nothing to write, `MaybeSendBytesFromBuffer()` never sets it back to `false` and the socket is stuck forever. The Fix: Simply move `m_send_ready = true` inside the block of `WriteReply()` where `m_send_mutex` is still held. This prevents the I/O thread from emptying the send buffer while the worker thread is setting the flag. Testing: To observe the race condition, revert the first commit `"http: prevent race condition between worker thread and I/O thread"` and run the unit test from the remainder of the branch. I like to see the logs: `test_bitcoin --log_level=all --run_test=httpserver_tests -- --printtoconsole --debug=http --debug=lock' The test will fail with a small probability. The socket will get stuck and the test will abort after a 60 second timeout. To garuntee the race condition loses and fail the test every time, slow down `WriteReply()` in the worker thread: ```diff diff --git a/src/httpserver.cpp b/src/httpserver.cpp index 99e30ff..b0c7b516d8 100644 --- a/src/httpserver.cpp +++ b/src/httpserver.cpp @@ -614,6 +614,7 @@ void HTTPRequest::WriteReply(HTTPStatusCode status, std::span<const std::byte> r } else { // Inform HTTPServer I/O that data is ready to be sent to this client // in the next loop iteration. + std::this_thread::sleep_for(500ms); m_client->m_send_ready = true; } ``` With the first commit (the fix) back in place, slowing down the worker thread like this won't fail the test. Bonus: The unit test is spread over three commits. First, a method of the socket testing setup is templated so a mock socket that intentionally raises an error can be inserted. The unit test added in that commit covers a race condition that was fixed in #35182 in response to https://github.com/bitcoin/bitcoin/pull/35182/changes#r3358889539 so we get the added benefit of covering an error path, and guaranteeing coverage of both "optimistic send" (directly from worker thread) and regular send (from a tick in the I/O loop thread). The next commit adds a worker thread to the unit test, at which point a race condition is possible but very unlikely because all requests are sent at once. Finally, we spread out the requests in the top commit and make the race condition much easier to catch. ACKs for top commit: janb84: crACK cd95a5a dergoegge: utACK cd95a5a theStack: Code-review ACK cd95a5a Tree-SHA512: 451982fd72724c4115e371fc6392605693d6c3207f00ffebcf027aae9253f7974b5b1165b9f46c91b5436d7fe60c7d27316fb0b79f729ab0bf8f32db2530075f

pinheadmz mentioned this pull request Jun 27, 2026

Intermittent RPC Timeout/Deadlock during large POST requests on Keep-Alive connections (TCP Zero Window) [was: Intermittent deadlock in p2p_orphan_handling.py", test_framework.authproxy.JSONRPCException: timeout] #34731

Closed

pinheadmz force-pushed the http-slow-sock branch from 617a5f1 to d3dd7bf Compare June 27, 2026 13:07

DrahtBot added the CI failed label Jun 27, 2026

pinheadmz force-pushed the http-slow-sock branch from d3dd7bf to f190c77 Compare June 27, 2026 13:23

DrahtBot removed the CI failed label Jun 27, 2026

fanquake added this to the 32.0 milestone Jun 27, 2026

DrahtBot mentioned this pull request Jun 28, 2026

Remove boost as a unit test runner #35587

Closed

dergoegge approved these changes Jun 29, 2026

View reviewed changes

janb84 reviewed Jun 29, 2026

View reviewed changes

pinheadmz force-pushed the http-slow-sock branch from f190c77 to 8fa4426 Compare June 29, 2026 14:00

pinheadmz added 4 commits June 29, 2026 10:09

test: introduce a worker thread in http socket error test

b98b10c

pinheadmz force-pushed the http-slow-sock branch from 8fa4426 to f595daf Compare June 29, 2026 14:32

janb84 reviewed Jun 29, 2026

View reviewed changes

DrahtBot requested a review from dergoegge June 29, 2026 18:32

dergoegge approved these changes Jun 30, 2026

View reviewed changes

theStack approved these changes Jul 3, 2026

View reviewed changes

fanquake merged commit 239d6c5 into bitcoin:master Jul 3, 2026
26 checks passed

pinheadmz mentioned this pull request Jul 8, 2026

http: check rpcallowip immediately after accepting connection #35592

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HTTPServer: Prevent race condition between worker thread and I/O thread#35614

HTTPServer: Prevent race condition between worker thread and I/O thread#35614
fanquake merged 4 commits into
bitcoin:masterfrom
pinheadmz:http-slow-sock

pinheadmz commented Jun 27, 2026

Uh oh!

DrahtBot commented Jun 27, 2026 •

edited

Loading

Uh oh!

DrahtBot commented Jun 27, 2026

Uh oh!

dergoegge left a comment

Uh oh!

janb84 left a comment

Uh oh!

janb84 Jun 29, 2026

Uh oh!

pinheadmz Jun 29, 2026

Uh oh!

pinheadmz commented Jun 29, 2026 •

edited

Loading

Uh oh!

janb84 left a comment

Uh oh!

dergoegge left a comment

Uh oh!

theStack left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Uh oh!

Conversation

pinheadmz commented Jun 27, 2026

Uh oh!

DrahtBot commented Jun 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Coverage & Benchmarks

Reviews

Conflicts

Uh oh!

DrahtBot commented Jun 27, 2026

Uh oh!

dergoegge left a comment

Choose a reason for hiding this comment

Uh oh!

janb84 left a comment

Choose a reason for hiding this comment

Uh oh!

janb84 Jun 29, 2026

Choose a reason for hiding this comment

Uh oh!

pinheadmz Jun 29, 2026

Choose a reason for hiding this comment

Uh oh!

pinheadmz commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

janb84 left a comment

Choose a reason for hiding this comment

Uh oh!

dergoegge left a comment

Choose a reason for hiding this comment

Uh oh!

theStack left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

DrahtBot commented Jun 27, 2026 •

edited

Loading

pinheadmz commented Jun 29, 2026 •

edited

Loading