Skip to content

Introduce waitTipChanged() mining interface, replace RPCNotifyBlockChange, drop CRPCSignals & g_best_block#30409

Merged
achow101 merged 12 commits into
bitcoin:masterfrom
Sjors:2024/07/wait-tip-changed
Sep 23, 2024
Merged

Introduce waitTipChanged() mining interface, replace RPCNotifyBlockChange, drop CRPCSignals & g_best_block#30409
achow101 merged 12 commits into
bitcoin:masterfrom
Sjors:2024/07/wait-tip-changed

Conversation

@Sjors

@Sjors Sjors commented Jul 8, 2024

Copy link
Copy Markdown
Member

This continues the work in #30200 so that a future Stratum v2 Template Provider (see #29432) can avoid accessing node internals. It needs to know when a new block arrives in order to push new templates to connected clients.

waitTipChanged() uses a new kernel notification notifications().m_tip_block_mutex, which this PR also introduces (a previous version used g_best_block).

In order to ensure the new method works as intended, the waitfornewblock, waitforblock and waitforblockheight RPC methods are refactored to use it. This allows removing RPCNotifyBlockChange.

There's a commit to add (direct) tests for the methods that are about to be refactored:

  • waitfornewblock was already implicitly tested by feature_shutdown.py.
  • waitforblockheight by feature_coinstatsindex.py and example_test.py

This PR renames getTipHash() to getTip() and returns a BlockRef (renamed from BlockKey) so that callers can use either the height or hash.

The later commits make trivial improvements to the waitfor* RPC calls (not needed for this PR).

The waitTipChanged() method could probably also be used for the longpoll functionality in getblocktemplate, but I'm a bit reluctant to touch that.

RPCServer::OnStarted no longer does anything and RPCServer::OnStopped merely prints a log statement. They were added in #5711 as a refactor. This PR drops them entirely.

Finally g_best_block is also dropped.

@DrahtBot

DrahtBot commented Jul 8, 2024

Copy link
Copy Markdown
Contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage

For detailed information about the code coverage, see the test coverage report.

Reviews

See the guideline for information on the review process.

Type Reviewers
ACK ryanofsky, TheCharlatan, achow101

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

Reviewers, this pull request conflicts with the following ones:

  • #30437 (multiprocess: add bitcoin-mine test program by ryanofsky)
  • #26593 (tracing: Only prepare tracepoint arguments when actually tracing by 0xB10C)
  • #19461 (multiprocess: Add bitcoin-gui -ipcconnect option by ryanofsky)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

@Sjors Sjors mentioned this pull request Jul 8, 2024
24 tasks
@Sjors Sjors force-pushed the 2024/07/wait-tip-changed branch from 7caba51 to b442f37 Compare July 9, 2024 06:47
@Sjors Sjors changed the title Introduce waitTipChanged() mining interface and replace RPCNotifyBlockChange Introduce waitTipChanged() mining interface, replace RPCNotifyBlockChange and drop CRPCSignals Jul 9, 2024
@Sjors Sjors force-pushed the 2024/07/wait-tip-changed branch from b442f37 to 4a2296f Compare July 9, 2024 07:54
@Sjors

Sjors commented Jul 9, 2024

Copy link
Copy Markdown
Member Author

Some of the functional tests that use the refactored RPC calls still fail under TSAN, so keeping this draft until that's resolved.

@Sjors Sjors force-pushed the 2024/07/wait-tip-changed branch from 4a2296f to 0fd7f52 Compare July 9, 2024 08:51
@Sjors

Sjors commented Jul 9, 2024

Copy link
Copy Markdown
Member Author

Fixed the deadlock. I ran into the same issue before when working on #29432: UpdateTip requires holding cs_main. It then takes g_best_block_mutex in order to announces the new block. So then lines after g_best_block_cv.wait_until should release g_best_block_mutex before trying to lock cs_main.

That's not a problem here; I just had to break out of the while loop instead of returning from it.

Back in the other PR this led to a deadlock if we simultaneously submitted a block and received it via p2p from the pool, which can happen in Stratum v2, see #29432 (comment)

@Sjors Sjors marked this pull request as ready for review July 9, 2024 08:56
@DrahtBot DrahtBot removed the CI failed label Jul 9, 2024
@Sjors Sjors force-pushed the 2024/07/wait-tip-changed branch from 0fd7f52 to 377c4b5 Compare July 11, 2024 11:55
Comment thread src/node/interfaces.cpp Outdated
ryanofsky added a commit to ryanofsky/bitcoin that referenced this pull request Jul 16, 2024
ryanofsky added a commit to ryanofsky/bitcoin that referenced this pull request Jul 17, 2024
ryanofsky added a commit to ryanofsky/bitcoin that referenced this pull request Jul 17, 2024
ryanofsky added a commit to ryanofsky/bitcoin that referenced this pull request Jul 17, 2024
Add interface changes from

bitcoin#30409
bitcoin#30356
bitcoin#30440
bitcoin#30443

This commit changes the C++ Mining interface without changing the corresponding
Capn'Proto interface. The capnp interface is updated the next commit.
ryanofsky added a commit to ryanofsky/bitcoin that referenced this pull request Jul 17, 2024
Add interface changes from

bitcoin#30409
bitcoin#30356
bitcoin#30440
bitcoin#30443

This commit updates the Cap'n Proto Mining interface after updating the C++
Mining interface last commit.
@Sjors Sjors force-pushed the 2024/07/wait-tip-changed branch from 966027b to 76ec301 Compare September 13, 2024 08:28
@Sjors

Sjors commented Sep 13, 2024

Copy link
Copy Markdown
Member Author

Trivial #include rebase after #30546.

@ryanofsky ryanofsky left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review ACK 76ec301c60e2530de7f58e2e4cea3d15c6a77cbc. Just simple rebase since last review

Sjors and others added 12 commits September 17, 2024 09:14
Also signal m_tip_block_cv when StopRPC is called, for
consistency with g_best_block_cv. This is handled in
StopRPC instead of OnRPCStopped() because the latter
is deleted in a later commit.

Co-authored-by: TheCharlatan <seb.kung@gmail.com>
Co-authored-by: Ryan Ofsky <ryan@ofsky.org>
Co-authored-by: TheCharlatan <seb.kung@gmail.com>
The waitforblock RPC method takes a hash argument and waits for that specific block.  The waitfornewblock waits for any new block. This commit fixes the documentation.
This refactoring commit uses the newly introduced waitTipChanged mining interface method to replace the RPCNotifyBlockChange mechanism.
They are no longer used for anything since RPCNotifyBlockChange was replaced with waitTipChanged() from the mining interface.
This removes the last remaining use of g_best_block by the RPC.
@Sjors

Sjors commented Sep 17, 2024

Copy link
Copy Markdown
Member Author

Rebased after #30440. Just some #include and using conflicts since that PR also touched the mining interface.

@ryanofsky ryanofsky left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review ACK 7942951. Just rebased since last review

@sedited sedited left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-ACK 7942951

@achow101

Copy link
Copy Markdown
Member

ACK 7942951

@maflcko maflcko left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

review ACK 7942951 🛋

Show signature

Signature:

untrusted comment: signature from minisign secret key on empty file; verify via: minisign -Vm "${path_to_any_empty_file}" -P RWTRmVTMeKV5noAMqVlsMugDDCyyTSbA3Re5AkUrhvLVln0tSaFWglOw -x "${path_to_this_whole_four_line_signature_blob}"
RUTRmVTMeKV5npGrKx1nqXCw5zeVHdtdYURB/KlyA/LMFgpNCs+SkW9a8N95d+U4AP1RJMi+krxU1A3Yux4bpwZNLvVBKy0wLgM=
trusted comment: review ACK 7942951e3fcc58f7db0059546d03be9ea243f1db 🛋
KRzS4gS3SXCvu+SqXrmOlO59h7EfpDm03wNYXbXyxcUpf1uSvRs0Wxsdg7n96gG9dwQPet+ZPAL6FR2cNKqBCw==

Comment thread src/rpc/server.cpp
WITH_LOCK(g_deadline_timers_mutex, deadlineTimers.clear());
DeleteAuthCookie();
g_rpcSignals.Stopped();
node::NodeContext& node = EnsureAnyNodeContext(context);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit in 7eccdaf: Any reason to use the throwing function? The internal error isn't caught anywhere, IIUC. So, the program will terminate either way. By using *Assert(AnyPtr(context)) here instead, at least a usable error message is printed.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I probably just copied similar code from another place :-)

Can make a followup, though *Assert(AnyPtr(context)) looks pretty scary.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You probably copied it from an RPC method implementation. There, it would be the right choice, because the http threads catch exceptions and then continue with the next RPC call.

However, here no exceptions are caught (and doing so wouldn't make sense, probably). The program will terminate either way, for example:

2024-09-24T10:38:06Z opencon thread exit
terminate called after throwing an instance of 'UniValue'
Aborted (core dumped)

However, by using Assert/assert, at least a source location is printed. If you don't like the assert, you can also use an if guard against a nullptr, but I am not sure if that is cleaner.

In any case, this is just a style nit.

Comment thread src/node/interfaces.cpp
Comment on lines +945 to +946
// std::chrono does not check against overflow
if (deadline < now) deadline = std::chrono::steady_clock::time_point::max();

@maflcko maflcko Sep 24, 2024

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

b94b27c: Can you explain the meaning of "overflow" in the context of double?

To clarify: Signed interger overflow is UB, so it can't happen. And overflow for double can't happen either, so this is dead code either way, unless I am missing something.

@Sjors Sjors Sep 24, 2024

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was in response to: #30409 (comment) where @ryanofsky wrote:

I'm also concerned about the deadline = now() + timeout assignment above because it seems like that will be an overflowed value when timeout is MillisecondsDouble::max() so the while loop condition might be invalid.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice catch. I believe this can be simplified to:

--- a/src/node/interfaces.cpp
+++ b/src/node/interfaces.cpp
@@ -941,9 +941,7 @@ public:
         // Interrupt check interval
         const MillisecondsDouble tick{1000};
         auto now{std::chrono::steady_clock::now()};
-        auto deadline = now + timeout;
-        // std::chrono does not check against overflow
-        if (deadline < now) deadline = std::chrono::steady_clock::time_point::max();
+        const auto deadline{now + timeout};
         {
             WAIT_LOCK(notifications().m_tip_block_mutex, lock);
             while ((notifications().m_tip_block == uint256() || notifications().m_tip_block == current_tip) && !chainman().m_interrupt) {

This previous code written to handle the case where now + timeout calculation would overflow, basically assuming both values were unsigned integer values and overflow behavior would be well defined. But since timeout is a floating point number where "overflow" would just result in inf being set, trying to deal with it this way doesn't make sense.

Mutex m_tip_block_mutex;
std::condition_variable m_tip_block_cv;
//! The block for which the last blockTip notification was received for.
uint256 m_tip_block;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs a GUARDED_BY(m_tip_block_mutex).

Adding it causes new warnings in validation_chainstate_tests.cpp. For ex:

src/test/validation_chainstate_tests.cpp:73:46: warning: reading variable 'm_tip_block' requires holding mutex 'm_node.notifications->m_tip_block_mutex' [-Wthread-safety-analysis]

I assume those particular accesses are harmless (maybe actually racy in the right conditions?), but future users may not be.

Related: It's a shame that the condvar/mutex leak out of here. Is there a reason not to add getter/waiter/notifier functions?

@Sjors

Sjors commented Sep 25, 2024

Copy link
Copy Markdown
Member Author

I'll work on a followup for the above things.

@maflcko

maflcko commented Sep 25, 2024

Copy link
Copy Markdown
Member

I'll work on a followup for the above things.

Already started working on this, because I needed it for other stuff.

Happy to drop mine, if you are done already.

@Sjors

Sjors commented Sep 25, 2024

Copy link
Copy Markdown
Member Author

@maflcko #30966

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants