-
Notifications
You must be signed in to change notification settings - Fork 39.2k
Fix startup failure with RLIM_INFINITY fd limits #34937
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -9,12 +9,14 @@ | |
|
|
||
| #include <sync.h> | ||
| #include <util/byte_units.h> // IWYU pragma: keep | ||
| #include <util/check.h> | ||
| #include <util/fs.h> | ||
| #include <util/log.h> | ||
| #include <util/syserror.h> | ||
|
|
||
| #include <cerrno> | ||
| #include <fstream> | ||
| #include <limits> | ||
| #include <map> | ||
| #include <memory> | ||
| #include <optional> | ||
|
|
@@ -152,27 +154,40 @@ bool TruncateFile(FILE* file, unsigned int length) | |
| #endif | ||
| } | ||
|
|
||
| /** | ||
| * this function tries to raise the file descriptor limit to the requested number. | ||
| * It returns the actual file descriptor limit (which may be more or less than nMinFD) | ||
| */ | ||
| int RaiseFileDescriptorLimit(int nMinFD) | ||
| int RaiseFileDescriptorLimit(int min_fd) | ||
| { | ||
| Assert(min_fd >= 0); | ||
| #if defined(WIN32) | ||
| return 2048; | ||
| #else | ||
| struct rlimit limitFD; | ||
| if (getrlimit(RLIMIT_NOFILE, &limitFD) != -1) { | ||
| if (limitFD.rlim_cur < (rlim_t)nMinFD) { | ||
| limitFD.rlim_cur = nMinFD; | ||
| if (limitFD.rlim_cur > limitFD.rlim_max) | ||
| // If the current soft limit is already higher, don't raise it | ||
| if (limitFD.rlim_cur != RLIM_INFINITY && std::cmp_less(limitFD.rlim_cur, min_fd)) { | ||
| const auto current_limit{limitFD.rlim_cur}; | ||
| static_assert(std::in_range<rlim_t>(std::numeric_limits<int>::max())); | ||
| limitFD.rlim_cur = static_cast<rlim_t>(min_fd); | ||
| // Don't raise soft limit beyond hard limit | ||
| if ((limitFD.rlim_max != RLIM_INFINITY) && (limitFD.rlim_cur > limitFD.rlim_max)) { | ||
| limitFD.rlim_cur = limitFD.rlim_max; | ||
| setrlimit(RLIMIT_NOFILE, &limitFD); | ||
| getrlimit(RLIMIT_NOFILE, &limitFD); | ||
| } | ||
| if (current_limit != limitFD.rlim_cur) { | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Does this change guard against anything new, or is it just there to not do redundant work?
Member
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think the latter, just avoiding an unnecessary write and read call. The original code had the same intention. |
||
| setrlimit(RLIMIT_NOFILE, &limitFD); | ||
| getrlimit(RLIMIT_NOFILE, &limitFD); | ||
| } | ||
| } | ||
| // Check the (possibly raised) current soft limit against the special | ||
| // value of RLIM_INFINITY. Some platforms implement this as the maximum | ||
| // uint64, others as int64 (-1). Avoid casting even if the return type | ||
| // is changed to uint64_t. We also cap unlikely but possible values | ||
| // that would overflow int. | ||
| if (limitFD.rlim_cur == RLIM_INFINITY || | ||
| std::cmp_greater_equal(limitFD.rlim_cur, std::numeric_limits<int>::max())) { | ||
| return std::numeric_limits<int>::max(); | ||
| } | ||
| return limitFD.rlim_cur; | ||
| return static_cast<int>(limitFD.rlim_cur); | ||
| } | ||
| return nMinFD; // getrlimit failed, assume it's fine | ||
| return min_fd; // getrlimit failed, assume it's fine | ||
| #endif | ||
| } | ||
|
|
||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
while auditing this line, i began to wonder: is there a practical "floor" we should be enforcing rather than allowing the process to attempt a start with values that guarantee a crash on non-Windows systems?
while manually incrementing limits during local functional tests, i realised that setting
min_fdto values between 0 and 10 consistently triggers a failure. specifically, if the environment provides fewer than 5 descriptors, the node cannot even initialise its basic logging or database handles before the OS kernel denies further requests.i observed that the functional test framework itself requires a slightly higher overhead to capture
stdout, but the core issue remains: a limit that is technically "non-negative" can still be physically impossible for a bitcoin node to survive.through manual isolation, i've identified two distinct failure thresholds:
bitcoindbinary fails to start entirely, as it cannot satisfy basic requirements for standard i/o and its primary log file/entropy source.proof of crash
the test i used.
my logs after the crash.
suggestion
rather than allowing the process to hit a
OSError: [Errno 24] Too many open filesmid-boot, we could perhaps treat this as a validation gate. it might be more robust to notify the user that the provided limit is insufficient for basic operation, providing a "fail-fast" mechanism rather than an ungraceful crash.given that the node fundamentally requires a small handful of descriptors just to "breathe" (logging, leveldb, entropy), do you think it is worth adjusting the c++ logic to cater for these very low-value edge cases, or is the current assertion sufficient for our purposes?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The point of an Assert is that we expect this to never crash. The caller needs to make sure no value
< 0is passed in. SinceRaiseFileDescriptorLimitis only called in one place, we know that is the case. If we ever add new call sites, then this assert would need to be re-evaluated.The number requested by the init code should be sufficient for node operations. It might be possible to get by with even less, but afaik there's no realistic environment where that's the case.
I guess the scenario you're describing is where we don't even get to
RaiseFileDescriptorLimit? I don't think that's worth worry about, assuming my assumption above is correct.Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
agreed!
prior to calling
RaiseFileDescriptorLimit, the logic ininit.cpp'sAppInitParameterInteractionfunction already aggregatesuser_max_connections,max_private, andmin_required_fds. this calculated sum (was 176 when i ranbitcoindthrough GDB on Linux) ensures the request is well above the physical floor required for basic stability.true!
the failure happens during initialisation before the
RaiseFileDescriptorLimitis even reached. I think I was perhaps being a bit too paranoid from an adversarial POV, imagining a scenario where a malicious actor gains enough access to suffocate the environment'sulimitto 8 or less, essentially DOSing the node's ability to even log its own failure. given how unlikely that environment is in practice, i agree it's not worth the extra complexity.thanks for the clarification!