Skip to content

fuzz: Abort when using global PRNG without re-seed#31486

Merged
ryanofsky merged 3 commits into
bitcoin:masterfrom
maflcko:2412-fuzz-rng-abort
Dec 17, 2024
Merged

fuzz: Abort when using global PRNG without re-seed#31486
ryanofsky merged 3 commits into
bitcoin:masterfrom
maflcko:2412-fuzz-rng-abort

Conversation

@maflcko

@maflcko maflcko commented Dec 13, 2024

Copy link
Copy Markdown
Member

This is the first step toward improving fuzz stability and determinism (#29018).

A fuzz target using the global test-only PRNG will now abort if the seed is re-used across fuzz inputs.

Also, temporarily add SeedRandomStateForTest(SeedRand::ZEROS) to all affected fuzz targets. This may slow down the libfuzzer leak detector, but it will disable itself after some time, or it can be disabled explicitly with -detect_leaks=0.

In a follow-up, each affected fuzz target can be stripped of the global random use and a local RandomMixin (or similar) can be added instead.

(Can be tested by removing any one of the re-seed calls and observing a fuzz abort)

@DrahtBot

DrahtBot commented Dec 13, 2024

Copy link
Copy Markdown
Contributor

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Code Coverage & Benchmarks

For details see: https://corecheck.dev/bitcoin/bitcoin/pulls/31486.

Reviews

See the guideline for information on the review process.

Type Reviewers
ACK hodlinator, dergoegge, marcofleon

If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update.

Conflicts

Reviewers, this pull request conflicts with the following ones:

  • #31481 (fuzz: Faster leak check, and SeedRand::ZEROS before every input by maflcko)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

@dergoegge

Copy link
Copy Markdown
Member

Concept ACK - Nice!

I kinda prefer this over #31481

@maflcko maflcko force-pushed the 2412-fuzz-rng-abort branch from 4444c20 to fae871e Compare December 13, 2024 13:15
@maflcko maflcko force-pushed the 2412-fuzz-rng-abort branch from fae871e to fa5c0da Compare December 13, 2024 13:24
@maflcko

maflcko commented Dec 13, 2024

Copy link
Copy Markdown
Member Author

(split into two commits to make review easier, no overall diff)

@marcofleon

Copy link
Copy Markdown
Contributor

Removed the call for a few targets and the expected behavior (message and crash) happens for libfuzzer but not for afl++. With afl++, removing the call leads to a very obvious slowdown for some of the targets.

For example for tx_package_eval, starting from an empty corpus, removing SeedRandomStateForTest(SeedRand::ZEROS) causes the target to go from ~5000 exec/s to ~7 exec/sec.

@marcofleon

Copy link
Copy Markdown
Contributor

Weird because test_one_input is called for AFL persistent mode so I don't see why it wouldn't also abort.

@maflcko

maflcko commented Dec 13, 2024

Copy link
Copy Markdown
Member Author

removing SeedRandomStateForTest(SeedRand::ZEROS) causes the target to go from ~5000 exec/s to ~7 exec/sec.

Interesting, are you sure it is not the other way round? Did you try with addition_overflow (or another simple target) as well? Does it also happen with the fixed inputs? Otherwise, it may just be the fuzz engine walking a path, like in #30644 (comment) ? What options did you compile with? What are the steps to reproduce, starting from a fresh install of the OS?

Comment thread src/test/fuzz/fuzz.cpp Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this needs to be a std::abort() (or similar), otherwise out of process fuzzers like afl will not catch this as a crash (by default at least)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You may be able to confirm this with AFL_CRASH_EXITCODE='1' (i.e. tell afl++ to treat exit(1) as a crash)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thx, fixed!

@dergoegge

Copy link
Copy Markdown
Member

the target to go from ~5000 exec/s to ~7 exec/sec

That is likely caused by persistent mode no longer working properly because of the std::exit, it basically degrades into non-persistent mode (i.e. re-forking every time it hits the std::exit).

@maflcko maflcko force-pushed the 2412-fuzz-rng-abort branch from fa5c0da to faebe6b Compare December 13, 2024 15:54

@hodlinator hodlinator left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Concept ACK fa5a1d524aa5647ca3a5f61c664adccd1904849e

Great to push fuzz test determinism forward.


Seems like SeedRandomStateForTest(SeedRand::ZEROS) should be removed from initialize() in fuzz.cpp?

Comment thread src/test/fuzz/fuzz.cpp Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this also be an abort(), or should the commit message be adjusted to say something else than "Abort"?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Commit fa5a1d524aa5647ca3a5f61c664adccd1904849e seems a bit too fragile overall, because it could detect false-positives. So I went ahead and dropped it for now. It should be enough if someone cherry-picks and runs it every release (6mo), or so.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I guess if a fuzz target only executes code blocks using randomness depending on the fuzz-data, this could have been annoying for short fuzz-runs.

Comment thread src/test/util/random.h Outdated
Comment on lines 31 to 32

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Try to adhere to symbol naming convention from developer-notes.md?

Suggested change
extern std::atomic<bool> seeded_g_prng_zero;
extern std::atomic<bool> used_g_prng;
extern std::atomic<bool> g_seeded_prng_zero;
extern std::atomic<bool> g_used_prng;

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thx, added g_.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should the _g_ in the middle of g_seeded_g_prng_zero and the other one still be kept?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, because this is only about the global prng, not about any local ones.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Would prefer g_seeded_global_prng_zero in that case, even though I admit it is verbose).

Comment thread src/test/fuzz/fuzz.cpp Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Naming + static

Suggested change
bool ever_used_g_prng{false};
static bool g_ever_used_prng{false};

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dropped commit

Comment thread src/test/util/random.cpp Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug in that it can be reset if called with FIXED_SEED afterwards:

Suggested change
seeded_g_prng_zero = seedtype == SeedRand::ZEROS;
seeded_g_prng_zero |= seedtype == SeedRand::ZEROS;

Wonder if it would be useful to also add checks against FIXED_SEED being used?

@maflcko maflcko Dec 16, 2024

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wonder if it would be useful to also add checks against FIXED_SEED being used?

Yes, this is what the code here is already doing. And your suggestion would remove the check (in some cases), no?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Current behavior with a test that calls both:

SeedRandomStateForTest(SeedRand::ZEROS);
SeedRandomStateForTest(SeedRand::FIXED_SEED);

Resulting in seeded_g_prng_zero not being set... which I guess is a way of detecting that something is wrong.

My suggestion would make seeded_g_prng_zero retain the fact that ZEROS had been used.

None of the above solutions detect this invalid case:

SeedRandomStateForTest(SeedRand::FIXED_SEED);
...fuzz target code using randomness...
SeedRandomStateForTest(SeedRand::ZEROS);

It would be more robust with my suggested change, + adding a separate seeded_g_prng_fixed bool which is (|=-)set when FIXED_SEED is used, so that we could error on that.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

adding a separate seeded_g_prng_fixed bool

No strong opinion. There is only one call to SeedRandomForTest(SeedRand::FIXED_SEED), so hopefully it won't be a problem going forward and the code doesn't have to be written and reviewed. Happy to reconsider, if other reviewers want it as well.

Comment thread src/test/fuzz/fuzz.cpp Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Bit of a naming clash with FuzzTarget::test_one_input. :/

FuzzTarget::test_one_input is a subpar as a field name, but if you don't want to touch that:

Suggested change
inline void test_one_input(FuzzBufferType buffer)
inline void execute_one_input(FuzzBufferType buffer)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving as-is for now, as I don't want to touch unrelated lines. test_one_input was already used before as the name, also it is used identically in https://github.com/bitcoin/bitcoin/pull/30882/files#diff-7f9d6992a63bfa29310450a7af7629ff9c324b275c4d94f102e8eca9daf118fd, so I'll also leave as-is to avoid (more) merge conflicts.

@maflcko maflcko force-pushed the 2412-fuzz-rng-abort branch from fa5a1d5 to fa2748c Compare December 16, 2024 09:59
@maflcko

maflcko commented Dec 16, 2024

Copy link
Copy Markdown
Member Author

Seems like SeedRandomStateForTest(SeedRand::ZEROS) should be removed from initialize() in fuzz.cpp?

No, why? Initialization happens before iterating over the inputs, so it is separate and isn't affected by this pull request.

@marcofleon

Copy link
Copy Markdown
Contributor

You think this same concept is worth doing for GetTime() and SetMockTime() as well? I'd be happy to implement it if so, or you could add it to this PR if that's easier.

@maflcko maflcko force-pushed the 2412-fuzz-rng-abort branch from fa2748c to fa9baad Compare December 16, 2024 13:56
@maflcko

maflcko commented Dec 16, 2024

Copy link
Copy Markdown
Member Author

Makes sense to extend the approach to other globals as well. I've moved it to a new module, so that the linter/check does not clutter the main fuzz.cpp file, if it is extended more. As for the (global) system time check, I presume it would fail, depending on the args that were used to fuzz (e.g. --printtoconsole=1), so I think I'll leave it as-is for now. However, I am happy to push a commit, if it works. Also, I am happy to review a follow-up pull. I can also look into opening one myself.

@DrahtBot

Copy link
Copy Markdown
Contributor

🚧 At least one of the CI tasks failed.
Debug: https://github.com/bitcoin/bitcoin/runs/34474894710

Hints

Try to run the tests locally, according to the documentation. However, a CI failure may still
happen due to a number of reasons, for example:

  • Possibly due to a silent merge conflict (the changes in this pull request being
    incompatible with the current code in the target branch). If so, make sure to rebase on the latest
    commit of the target branch.

  • A sanitizer issue, which can only be found by compiling with the sanitizer and running the
    affected test.

  • An intermittent issue.

Leave a comment here, if you need help tracking down a confusing failure.

@hodlinator hodlinator left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code reviewed fa18acb


Seems like SeedRandomStateForTest(SeedRand::ZEROS) should be removed from initialize() in fuzz.cpp?

No, why? Initialization happens before iterating over the inputs, so it is separate and isn't affected by this pull request.

Yes, but does it still provide any value when the fuzz targets themselves are now pushed to call it?

Comment thread src/test/fuzz/fuzz.cpp Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I guess if a fuzz target only executes code blocks using randomness depending on the fuzz-data, this could have been annoying for short fuzz-runs.

Comment on lines +11 to +12
#include <optional>
#include <string>

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Unused

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thx. Will fix if I have to re-push. In the future, it would be better to check this in the CI, see #31308

Comment thread src/random.cpp

void GetRandBytes(Span<unsigned char> bytes) noexcept
{
g_used_g_prng = true;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skip in production

Suggested change
g_used_g_prng = true;
if constexpr (G_FUZZING) {
g_used_g_prng = true;
}

or use #ifdef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION.

Maybe an optimizer would remove it as a dead-store, but better not to rely on that.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a (micro-)benchmark that would improve?

Also, this would make it harder to use in other tests, such as the unit tests.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is about excluding test-only code from production (currently fuzz-only since check_globals.h/cpp is in fuzz/util/). Not sure why that would be undesirable. Even though the conditional adds verbosity, it adds context and decreases need for comments.

Might be worth introducing a TESTS_ENABLED #define... but I'm guessing that has been discussed before?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might be worth introducing a TESTS_ENABLED #define... but I'm guessing that has been discussed before?

I don't think it has. Maybe a separate issue or pull request can be created?

Whatever is done here also applies to the function immediately above, which is unrelated, so I'll leave this as-is for now.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting that it hasn't really been discussed. There are trade-offs to adding it, but yeah, might experiment with adding it and create an issue.

Correct, MakeRandDeterministicDANGEROUS() should not exist unless TESTS_ENABLED was #defined (in that hypothetical scenario).

g_seeded_g_prng_zero = false;
}
~CheckGlobalsImpl()
{

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should skip this if fuzz target throws an exception.

Suggested change
{
{
if (std::uncaught_exceptions() > 0) {
std::cerr << "Skipping checking of PRNG globals as exception is being thrown.\n";
return;
}

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems more important to make the code reproducible and stable before extended fuzzing. Also, if the crash is reproducible, it will be re-detected anyway when the code is fixed. So I think the extra code isn't needed.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm.. I guess it should pretty much never happen ™️ as the exception would have to be thrown before the seeding of the RNG at the beginning of a test.

Comment thread src/test/util/random.h Outdated
Comment on lines 31 to 32

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should the _g_ in the middle of g_seeded_g_prng_zero and the other one still be kept?

Comment thread src/test/util/random.cpp Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Current behavior with a test that calls both:

SeedRandomStateForTest(SeedRand::ZEROS);
SeedRandomStateForTest(SeedRand::FIXED_SEED);

Resulting in seeded_g_prng_zero not being set... which I guess is a way of detecting that something is wrong.

My suggestion would make seeded_g_prng_zero retain the fact that ZEROS had been used.

None of the above solutions detect this invalid case:

SeedRandomStateForTest(SeedRand::FIXED_SEED);
...fuzz target code using randomness...
SeedRandomStateForTest(SeedRand::ZEROS);

It would be more robust with my suggested change, + adding a separate seeded_g_prng_fixed bool which is (|=-)set when FIXED_SEED is used, so that we could error on that.

@maflcko

maflcko commented Dec 16, 2024

Copy link
Copy Markdown
Member Author

Seems like SeedRandomStateForTest(SeedRand::ZEROS) should be removed from initialize() in fuzz.cpp?

No, why? Initialization happens before iterating over the inputs, so it is separate and isn't affected by this pull request.

Yes, but does it still provide any value when the fuzz targets themselves are now pushed to call it?

Yes, because initialization should be deterministic as well. Otherwise, there could be stability or determinism issues as well.

@hodlinator hodlinator left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like SeedRandomStateForTest(SeedRand::ZEROS) should be removed from initialize() in fuzz.cpp?

..

[...] initialization should be deterministic as well. [...]

Ah, sorry for not getting it the first time.

Current code:

void initialize()
{
// By default, make the RNG deterministic with a fixed seed. This will affect all
// randomness during the fuzz test, except:
// - GetStrongRandBytes(), which is used for the creation of private key material.
// - Creating a BasicTestingSetup or derived class will switch to a random seed.
SeedRandomStateForTest(SeedRand::ZEROS);

BasicTestingSetup which is used during several fuzz initialization functions has the only such call you mentioned in another thread:

SeedRandomForTest(SeedRand::FIXED_SEED);

Should that call be disabled for fuzz-tests?

Comment thread src/test/util/random.h Outdated
Comment on lines 31 to 32

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Would prefer g_seeded_global_prng_zero in that case, even though I admit it is verbose).

Comment thread src/random.cpp

void GetRandBytes(Span<unsigned char> bytes) noexcept
{
g_used_g_prng = true;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is about excluding test-only code from production (currently fuzz-only since check_globals.h/cpp is in fuzz/util/). Not sure why that would be undesirable. Even though the conditional adds verbosity, it adds context and decreases need for comments.

Might be worth introducing a TESTS_ENABLED #define... but I'm guessing that has been discussed before?

g_seeded_g_prng_zero = false;
}
~CheckGlobalsImpl()
{

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm.. I guess it should pretty much never happen ™️ as the exception would have to be thrown before the seeding of the RNG at the beginning of a test.

@hodlinator

Copy link
Copy Markdown
Contributor

You think this same concept is worth doing for GetTime() and SetMockTime() as well? I'd be happy to implement it if so, or you could add it to this PR if that's easier.

Nice idea!
Has anyone discussed the possibility of adding a linter to catch rogue direct calls to std::chrono::system_clock::now() etc?

@marcofleon

Copy link
Copy Markdown
Contributor

Has anyone discussed the possibility of adding a linter to catch rogue direct calls to std::chrono::system_clock::now() etc?

I'm working on adding it on top of this PR, will post a branch here soon. A couple of the targets are giving me a bit of trouble... but I'll figure it out🧐

@maflcko

maflcko commented Dec 17, 2024

Copy link
Copy Markdown
Member Author

Should that call be disabled for fuzz-tests?

It could be. Either way is harmless, so I'll leave it as-is for now.

Edit: Pushed a commit to clarify that only ZEROS is allowed in fuzz tests.

@maflcko maflcko force-pushed the 2412-fuzz-rng-abort branch from 2222afe to fae63bf Compare December 17, 2024 07:46
@hodlinator

Copy link
Copy Markdown
Contributor

While I do much appreciate the intent behind adding commit fae63bf, I'm worried that it will cause issues in CMake configurations such as the "dev-mode"-preset which includes both fuzzing, unit tests, and a lot of other stuff. Having

if constexpr (G_FUZZING)

(or the negation of that) would cause unit tests to no longer use FIXED_SEEDS when fuzzing is also enabled.

Was instead thinking the fuzz-target initialization-functions would pass down a parameter into the BasicTestingSetup-ctor (maybe a new field in TestOpts?).

For the change inside SeedRandomStateForTest(), adding a dynamic g_seeded_g_prng_fixed_seed that can later be checked when fuzzing would enable unit tests to still operate in that mode.

(My suggestion would just have needlessly set g_used_g_prng in these builds when running unit tests).

I'm happy to explore implementing a dynamic version of that commit as outlined above if you don't feel like doing it but still might approve of it.

@maflcko

maflcko commented Dec 17, 2024

Copy link
Copy Markdown
Member Author

(or the negation of that) would cause unit tests to no longer use FIXED_SEEDS when fuzzing is also enabled.

No, it is only possible to run fuzz tests with G_FUZZING, and unit tests without it. This pull request is not changing that.

In any case, I don't really care about the commit. I am happy to drop it, or any other commit in this pull request and review a follow-up.

@hodlinator hodlinator left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK fae63bf

Makes fuzzing more deterministic through re-seeding the PRNG with the same value before testing every input. Might increase the efficiency of fuzzers in covering more code paths for some targets.

Tested removing SeedRandomStateForTest(SeedRand::ZEROS) from src/test/fuzz/process_message.cpp, ran the fuzz target and got the expected error output.

I see no remaining blockers.


No, it is only possible to run fuzz tests with G_FUZZING, and unit tests without it. This pull request is not changing that.

Ah, I was conflating BUILD_FOR_FUZZING with BUILD_FUZZ_BINARY in the "dev-mode"-preset. The unit test binary is not built when BUILD_FOR_FUZZING is set:

bitcoin/CMakeLists.txt

Lines 211 to 223 in fae63bf

if(BUILD_FOR_FUZZING)
message(WARNING "BUILD_FOR_FUZZING=ON will disable all other targets and force BUILD_FUZZ_BINARY=ON.")
set(BUILD_DAEMON OFF)
set(BUILD_CLI OFF)
set(BUILD_TX OFF)
set(BUILD_UTIL OFF)
set(BUILD_UTIL_CHAINSTATE OFF)
set(BUILD_KERNEL_LIB OFF)
set(BUILD_WALLET_TOOL OFF)
set(BUILD_GUI OFF)
set(ENABLE_EXTERNAL_SIGNER OFF)
set(WITH_ZMQ OFF)
set(BUILD_TESTS OFF)

@DrahtBot DrahtBot requested a review from dergoegge December 17, 2024 12:41
@maflcko maflcko requested a review from marcofleon December 17, 2024 13:05

@dergoegge dergoegge left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

utACK fae63bf

@marcofleon marcofleon left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested ACK fae63bf

The CheckGlobals module is a good addition, makes it easy to extend for other globals. I'll open a PR that checks for system time calls after this is merged.

@ryanofsky ryanofsky merged commit a60d570 into bitcoin:master Dec 17, 2024

@maflcko maflcko left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found a small issue after self re-review, sorry for missing it.

.extra_args = {"-testactivationheight=bip34@2"},
},
};
SeedRandomStateForTest(SeedRand::ZEROS); // Can not be done before test_setup

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, with the third commit (fae63bf), this looks now wrong. I can and must be done before test_setup.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in #31521

@maflcko maflcko deleted the 2412-fuzz-rng-abort branch December 18, 2024 08:12
Comment on lines +111 to +113
if constexpr (!G_FUZZING) {
SeedRandomForTest(SeedRand::FIXED_SEED);
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In commit "fuzz: Clarify that only SeedRandomStateForTest(SeedRand::ZEROS) is allowed" (fae63bf)

What's are reasons for not writing this as:

SeedRandomForTest(G_FUZZING ? SeedRand::ZEROS : SeedRand::FIXED_SEED);

It seems like this could eliminate boilerplate from fuzz tests, and also make the test setup behavior more intuitive: always setting a deterministic seed instead of sometimes setting one and sometimes not.

(Asking after this PR was already merged in context of followup #31521, which I'm finding a little confusing)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's are reasons for not writing this as:

SeedRandomForTest(G_FUZZING ? SeedRand::ZEROS : SeedRand::FIXED_SEED);

It seems like this could eliminate boilerplate from fuzz tests, and also make the test setup behavior more intuitive: always setting a deterministic seed instead of sometimes setting one and sometimes not.

Sure, this should be fine as well and be more intuitive.

However, I don't think it affects any boilerplate. In fuzzing the initialization phase is different from the execution phases, and each phase needs to seed the global PRNG with a constant (or something derived from the fuzz input in the execution phase). So the boilerplate can't be removed, see also the previous discussion #31486 (comment)

In fact, the boilerplate added in this pull is desirable, because it acts as a "todo" list of places where global randomness can be replaced by a local (fast) random context, if possible.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re: #31486 (comment)

So the boilerplate can't be removed

To clarify, in the context of #31521 specifically, I think the change I'm suggesting would have prevented the nondeterminism there, and would avoid the need for a SeedRandomStateForTest(SeedRand::ZEROS); call at the top of the utxo_total_supply test.

But the utxo_total_supply test is just an unusual test, because most fuzz tests use test setup objects that get shared across fuzz inputs, while utxo_total_supply creates a new test setup object for each input.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants