fuzz: Abort when using global PRNG without re-seed#31486
Conversation
|
The following sections might be updated with supplementary metadata relevant to reviewers and maintainers. Code Coverage & BenchmarksFor details see: https://corecheck.dev/bitcoin/bitcoin/pulls/31486. ReviewsSee the guideline for information on the review process.
If your review is incorrectly listed, please react with 👎 to this comment and the bot will ignore it on the next update. ConflictsReviewers, this pull request conflicts with the following ones:
If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first. |
|
Concept ACK - Nice! I kinda prefer this over #31481 |
4444c20 to
fae871e
Compare
fae871e to
fa5c0da
Compare
|
(split into two commits to make review easier, no overall diff) |
|
Removed the call for a few targets and the expected behavior (message and crash) happens for libfuzzer but not for afl++. With afl++, removing the call leads to a very obvious slowdown for some of the targets. For example for |
|
Weird because |
Interesting, are you sure it is not the other way round? Did you try with |
There was a problem hiding this comment.
this needs to be a std::abort() (or similar), otherwise out of process fuzzers like afl will not catch this as a crash (by default at least)
There was a problem hiding this comment.
You may be able to confirm this with AFL_CRASH_EXITCODE='1' (i.e. tell afl++ to treat exit(1) as a crash)
That is likely caused by persistent mode no longer working properly because of the |
fa5c0da to
faebe6b
Compare
hodlinator
left a comment
There was a problem hiding this comment.
Concept ACK fa5a1d524aa5647ca3a5f61c664adccd1904849e
Great to push fuzz test determinism forward.
Seems like SeedRandomStateForTest(SeedRand::ZEROS) should be removed from initialize() in fuzz.cpp?
There was a problem hiding this comment.
Should this also be an abort(), or should the commit message be adjusted to say something else than "Abort"?
There was a problem hiding this comment.
Commit fa5a1d524aa5647ca3a5f61c664adccd1904849e seems a bit too fragile overall, because it could detect false-positives. So I went ahead and dropped it for now. It should be enough if someone cherry-picks and runs it every release (6mo), or so.
There was a problem hiding this comment.
Yeah, I guess if a fuzz target only executes code blocks using randomness depending on the fuzz-data, this could have been annoying for short fuzz-runs.
There was a problem hiding this comment.
nit: Try to adhere to symbol naming convention from developer-notes.md?
| extern std::atomic<bool> seeded_g_prng_zero; | |
| extern std::atomic<bool> used_g_prng; | |
| extern std::atomic<bool> g_seeded_prng_zero; | |
| extern std::atomic<bool> g_used_prng; |
There was a problem hiding this comment.
Should the _g_ in the middle of g_seeded_g_prng_zero and the other one still be kept?
There was a problem hiding this comment.
Yes, because this is only about the global prng, not about any local ones.
There was a problem hiding this comment.
(Would prefer g_seeded_global_prng_zero in that case, even though I admit it is verbose).
There was a problem hiding this comment.
nit: Naming + static
| bool ever_used_g_prng{false}; | |
| static bool g_ever_used_prng{false}; |
There was a problem hiding this comment.
Bug in that it can be reset if called with FIXED_SEED afterwards:
| seeded_g_prng_zero = seedtype == SeedRand::ZEROS; | |
| seeded_g_prng_zero |= seedtype == SeedRand::ZEROS; |
Wonder if it would be useful to also add checks against FIXED_SEED being used?
There was a problem hiding this comment.
Wonder if it would be useful to also add checks against
FIXED_SEEDbeing used?
Yes, this is what the code here is already doing. And your suggestion would remove the check (in some cases), no?
There was a problem hiding this comment.
Current behavior with a test that calls both:
SeedRandomStateForTest(SeedRand::ZEROS);
SeedRandomStateForTest(SeedRand::FIXED_SEED);Resulting in seeded_g_prng_zero not being set... which I guess is a way of detecting that something is wrong.
My suggestion would make seeded_g_prng_zero retain the fact that ZEROS had been used.
None of the above solutions detect this invalid case:
SeedRandomStateForTest(SeedRand::FIXED_SEED);
...fuzz target code using randomness...
SeedRandomStateForTest(SeedRand::ZEROS);It would be more robust with my suggested change, + adding a separate seeded_g_prng_fixed bool which is (|=-)set when FIXED_SEED is used, so that we could error on that.
There was a problem hiding this comment.
adding a separate
seeded_g_prng_fixedbool
No strong opinion. There is only one call to SeedRandomForTest(SeedRand::FIXED_SEED), so hopefully it won't be a problem going forward and the code doesn't have to be written and reviewed. Happy to reconsider, if other reviewers want it as well.
There was a problem hiding this comment.
nit: Bit of a naming clash with FuzzTarget::test_one_input. :/
FuzzTarget::test_one_input is a subpar as a field name, but if you don't want to touch that:
| inline void test_one_input(FuzzBufferType buffer) | |
| inline void execute_one_input(FuzzBufferType buffer) |
There was a problem hiding this comment.
Leaving as-is for now, as I don't want to touch unrelated lines. test_one_input was already used before as the name, also it is used identically in https://github.com/bitcoin/bitcoin/pull/30882/files#diff-7f9d6992a63bfa29310450a7af7629ff9c324b275c4d94f102e8eca9daf118fd, so I'll also leave as-is to avoid (more) merge conflicts.
fa5a1d5 to
fa2748c
Compare
No, why? Initialization happens before iterating over the inputs, so it is separate and isn't affected by this pull request. |
|
You think this same concept is worth doing for |
fa2748c to
fa9baad
Compare
|
Makes sense to extend the approach to other globals as well. I've moved it to a new module, so that the linter/check does not clutter the main |
fa9baad to
fa18acb
Compare
|
🚧 At least one of the CI tasks failed. HintsTry to run the tests locally, according to the documentation. However, a CI failure may still
Leave a comment here, if you need help tracking down a confusing failure. |
hodlinator
left a comment
There was a problem hiding this comment.
Code reviewed fa18acb
Seems like
SeedRandomStateForTest(SeedRand::ZEROS)should be removed frominitialize()in fuzz.cpp?No, why? Initialization happens before iterating over the inputs, so it is separate and isn't affected by this pull request.
Yes, but does it still provide any value when the fuzz targets themselves are now pushed to call it?
There was a problem hiding this comment.
Yeah, I guess if a fuzz target only executes code blocks using randomness depending on the fuzz-data, this could have been annoying for short fuzz-runs.
| #include <optional> | ||
| #include <string> |
There was a problem hiding this comment.
thx. Will fix if I have to re-push. In the future, it would be better to check this in the CI, see #31308
|
|
||
| void GetRandBytes(Span<unsigned char> bytes) noexcept | ||
| { | ||
| g_used_g_prng = true; |
There was a problem hiding this comment.
Skip in production
| g_used_g_prng = true; | |
| if constexpr (G_FUZZING) { | |
| g_used_g_prng = true; | |
| } |
or use #ifdef FUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION.
Maybe an optimizer would remove it as a dead-store, but better not to rely on that.
There was a problem hiding this comment.
Is there a (micro-)benchmark that would improve?
Also, this would make it harder to use in other tests, such as the unit tests.
There was a problem hiding this comment.
This is about excluding test-only code from production (currently fuzz-only since check_globals.h/cpp is in fuzz/util/). Not sure why that would be undesirable. Even though the conditional adds verbosity, it adds context and decreases need for comments.
Might be worth introducing a TESTS_ENABLED #define... but I'm guessing that has been discussed before?
There was a problem hiding this comment.
Might be worth introducing a
TESTS_ENABLED#define... but I'm guessing that has been discussed before?
I don't think it has. Maybe a separate issue or pull request can be created?
Whatever is done here also applies to the function immediately above, which is unrelated, so I'll leave this as-is for now.
There was a problem hiding this comment.
Interesting that it hasn't really been discussed. There are trade-offs to adding it, but yeah, might experiment with adding it and create an issue.
Correct, MakeRandDeterministicDANGEROUS() should not exist unless TESTS_ENABLED was #defined (in that hypothetical scenario).
| g_seeded_g_prng_zero = false; | ||
| } | ||
| ~CheckGlobalsImpl() | ||
| { |
There was a problem hiding this comment.
Should skip this if fuzz target throws an exception.
| { | |
| { | |
| if (std::uncaught_exceptions() > 0) { | |
| std::cerr << "Skipping checking of PRNG globals as exception is being thrown.\n"; | |
| return; | |
| } | |
There was a problem hiding this comment.
It seems more important to make the code reproducible and stable before extended fuzzing. Also, if the crash is reproducible, it will be re-detected anyway when the code is fixed. So I think the extra code isn't needed.
There was a problem hiding this comment.
Hm.. I guess it should pretty much never happen ™️ as the exception would have to be thrown before the seeding of the RNG at the beginning of a test.
There was a problem hiding this comment.
Should the _g_ in the middle of g_seeded_g_prng_zero and the other one still be kept?
There was a problem hiding this comment.
Current behavior with a test that calls both:
SeedRandomStateForTest(SeedRand::ZEROS);
SeedRandomStateForTest(SeedRand::FIXED_SEED);Resulting in seeded_g_prng_zero not being set... which I guess is a way of detecting that something is wrong.
My suggestion would make seeded_g_prng_zero retain the fact that ZEROS had been used.
None of the above solutions detect this invalid case:
SeedRandomStateForTest(SeedRand::FIXED_SEED);
...fuzz target code using randomness...
SeedRandomStateForTest(SeedRand::ZEROS);It would be more robust with my suggested change, + adding a separate seeded_g_prng_fixed bool which is (|=-)set when FIXED_SEED is used, so that we could error on that.
Yes, because initialization should be deterministic as well. Otherwise, there could be stability or determinism issues as well. |
hodlinator
left a comment
There was a problem hiding this comment.
Seems like
SeedRandomStateForTest(SeedRand::ZEROS)should be removed frominitialize()in fuzz.cpp?..
[...] initialization should be deterministic as well. [...]
Ah, sorry for not getting it the first time.
Current code:
bitcoin/src/test/fuzz/fuzz.cpp
Lines 111 to 117 in fa18acb
BasicTestingSetup which is used during several fuzz initialization functions has the only such call you mentioned in another thread:
bitcoin/src/test/util/setup_common.cpp
Line 142 in fa18acb
Should that call be disabled for fuzz-tests?
There was a problem hiding this comment.
(Would prefer g_seeded_global_prng_zero in that case, even though I admit it is verbose).
|
|
||
| void GetRandBytes(Span<unsigned char> bytes) noexcept | ||
| { | ||
| g_used_g_prng = true; |
There was a problem hiding this comment.
This is about excluding test-only code from production (currently fuzz-only since check_globals.h/cpp is in fuzz/util/). Not sure why that would be undesirable. Even though the conditional adds verbosity, it adds context and decreases need for comments.
Might be worth introducing a TESTS_ENABLED #define... but I'm guessing that has been discussed before?
| g_seeded_g_prng_zero = false; | ||
| } | ||
| ~CheckGlobalsImpl() | ||
| { |
There was a problem hiding this comment.
Hm.. I guess it should pretty much never happen ™️ as the exception would have to be thrown before the seeding of the RNG at the beginning of a test.
Nice idea! |
I'm working on adding it on top of this PR, will post a branch here soon. A couple of the targets are giving me a bit of trouble... but I'll figure it out🧐 |
It could be. Either way is harmless, so I'll leave it as-is for now. Edit: Pushed a commit to clarify that only ZEROS is allowed in fuzz tests. |
2222afe to
fae63bf
Compare
|
While I do much appreciate the intent behind adding commit fae63bf, I'm worried that it will cause issues in CMake configurations such as the "dev-mode"-preset which includes both fuzzing, unit tests, and a lot of other stuff. Having (or the negation of that) would cause unit tests to no longer use Was instead thinking the fuzz-target initialization-functions would pass down a parameter into the For the change inside (My suggestion would just have needlessly set I'm happy to explore implementing a dynamic version of that commit as outlined above if you don't feel like doing it but still might approve of it. |
No, it is only possible to run fuzz tests with G_FUZZING, and unit tests without it. This pull request is not changing that. In any case, I don't really care about the commit. I am happy to drop it, or any other commit in this pull request and review a follow-up. |
hodlinator
left a comment
There was a problem hiding this comment.
ACK fae63bf
Makes fuzzing more deterministic through re-seeding the PRNG with the same value before testing every input. Might increase the efficiency of fuzzers in covering more code paths for some targets.
Tested removing SeedRandomStateForTest(SeedRand::ZEROS) from src/test/fuzz/process_message.cpp, ran the fuzz target and got the expected error output.
I see no remaining blockers.
No, it is only possible to run fuzz tests with G_FUZZING, and unit tests without it. This pull request is not changing that.
Ah, I was conflating BUILD_FOR_FUZZING with BUILD_FUZZ_BINARY in the "dev-mode"-preset. The unit test binary is not built when BUILD_FOR_FUZZING is set:
Lines 211 to 223 in fae63bf
marcofleon
left a comment
There was a problem hiding this comment.
Tested ACK fae63bf
The CheckGlobals module is a good addition, makes it easy to extend for other globals. I'll open a PR that checks for system time calls after this is merged.
maflcko
left a comment
There was a problem hiding this comment.
Found a small issue after self re-review, sorry for missing it.
| .extra_args = {"-testactivationheight=bip34@2"}, | ||
| }, | ||
| }; | ||
| SeedRandomStateForTest(SeedRand::ZEROS); // Can not be done before test_setup |
There was a problem hiding this comment.
Actually, with the third commit (fae63bf), this looks now wrong. I can and must be done before test_setup.
| if constexpr (!G_FUZZING) { | ||
| SeedRandomForTest(SeedRand::FIXED_SEED); | ||
| } |
There was a problem hiding this comment.
In commit "fuzz: Clarify that only SeedRandomStateForTest(SeedRand::ZEROS) is allowed" (fae63bf)
What's are reasons for not writing this as:
SeedRandomForTest(G_FUZZING ? SeedRand::ZEROS : SeedRand::FIXED_SEED);It seems like this could eliminate boilerplate from fuzz tests, and also make the test setup behavior more intuitive: always setting a deterministic seed instead of sometimes setting one and sometimes not.
(Asking after this PR was already merged in context of followup #31521, which I'm finding a little confusing)
There was a problem hiding this comment.
What's are reasons for not writing this as:
SeedRandomForTest(G_FUZZING ? SeedRand::ZEROS : SeedRand::FIXED_SEED);It seems like this could eliminate boilerplate from fuzz tests, and also make the test setup behavior more intuitive: always setting a deterministic seed instead of sometimes setting one and sometimes not.
Sure, this should be fine as well and be more intuitive.
However, I don't think it affects any boilerplate. In fuzzing the initialization phase is different from the execution phases, and each phase needs to seed the global PRNG with a constant (or something derived from the fuzz input in the execution phase). So the boilerplate can't be removed, see also the previous discussion #31486 (comment)
In fact, the boilerplate added in this pull is desirable, because it acts as a "todo" list of places where global randomness can be replaced by a local (fast) random context, if possible.
There was a problem hiding this comment.
re: #31486 (comment)
So the boilerplate can't be removed
To clarify, in the context of #31521 specifically, I think the change I'm suggesting would have prevented the nondeterminism there, and would avoid the need for a SeedRandomStateForTest(SeedRand::ZEROS); call at the top of the utxo_total_supply test.
But the utxo_total_supply test is just an unusual test, because most fuzz tests use test setup objects that get shared across fuzz inputs, while utxo_total_supply creates a new test setup object for each input.
This is the first step toward improving fuzz stability and determinism (#29018).
A fuzz target using the global test-only PRNG will now abort if the seed is re-used across fuzz inputs.
Also, temporarily add
SeedRandomStateForTest(SeedRand::ZEROS)to all affected fuzz targets. This may slow down the libfuzzer leak detector, but it will disable itself after some time, or it can be disabled explicitly with-detect_leaks=0.In a follow-up, each affected fuzz target can be stripped of the global random use and a local
RandomMixin(or similar) can be added instead.(Can be tested by removing any one of the re-seed calls and observing a fuzz abort)