Optimize beacon API (REST and gRPC) regarding endpoints needing to iterate over the whole validator set. by nalepae · Pull Request #16838 · OffchainLabs/prysm

nalepae · 2026-05-24T19:49:35Z

What type of PR is this?
Bug fix and optimization

What does this PR do? Why is it needed?
Some of the beacon endpoints currently wastefully materialize the entire validator set (~2.3M validators on mainnet) with the state.Validators function before filtering interesting validators. The call to state.Validators is not free at all, especially on mainnet, and, because Ethereum doesn't currently have validator index re-uses, this call will become more and more expensive to do.

This PR implements the "yield" iterations for these endpoints, allowing them to filter "on the flight" validator they want to keep and thus preventing wasteful costly heap allocation.

Notable example of gain: /eth/v1/beacon/states/{state_id}/validators?status=withdrawal_possible.
Previously, all the validators where materialized in the node heap (~2.3M on mainnet). With this PR, only the exited slashed validators are materialized (1442 on mainnet).

This pull requests:

Fixes the listed issue
Applies the optimization avoiding materializing the whole validator set (~2.3M validators on mainnet) when possible, thanks to the function iterator golang feature.

Which issues(s) does this PR fix?

/prysm/v1/validators/head/active_set_changes returns the full active validators set, instead of only those that became activated during the epoch #16842

Other notes for review

Important

Please read commit by commit, with commit messages, since they contain useful information to understand the code changes.

Note

Two calls to ValidatorsReadOnly remain in hdiff. It may probably be interesting to remove them, but it will be done (if needed) in an other PR.
EDIT: (Seen with @potuz on Prysm internal messaging system: Will be done in a next PR.)

Acknowledgements

I have read CONTRIBUTING.md.
I have included a uniquely named changelog fragment file.
I have added a description with sufficient context for reviewers to understand this PR.
I have tested that my changes work as expected and I added a testing plan to the PR description (if applicable).

Previous implementation was returning ALL active validators, contrary to what is stated by the godoc.

…f an `int`. Rationale: Most of callers had to do this cast. Now the cast is directly done in the function, which is simpler for callers.

This (very slightly) adds some complexity to the implementation of `ValidatorsReadOnlySeq` (which is done only once), but it simplifies the call sites (which are numerous).

…dators (mainnet) heap allocation by using `requestedState.ValidatorsReadOnlySeq` instead of `requestedState.Validators`.

…over validators as an input instead of the full validators slice. It avoids a ~2.3M validators (mainnet) materialization.

…): Avoid full validators list (2.3M on mainnet) materialization.

…e): Avoid the full validators list (~2.3M on mainnet) materialization.

…list materialization.

potuz

LGTM

potuz · 2026-05-26T13:35:38Z

 			bal.ActiveCurrentEpoch, err = math.Add64(bal.ActiveCurrentEpoch, val.EffectiveBalance())
 			if err != nil {
-				return err
+				return nil, nil, fmt.Errorf("add 64: %w", err)


I guess Claude did not understand what the function Add64 does :)

Haha no it was not claude but me.
I always follow this convention:
https://github.com/uber-go/guide/blob/master/style.md#error-wrapping

But yes in this case in not so clear. Will fix

Fixed in 817ff4d.

potuz · 2026-05-26T13:36:14Z

 			bal.ActivePrevEpoch, err = math.Add64(bal.ActivePrevEpoch, val.EffectiveBalance())
 			if err != nil {
-				return err
+				return nil, nil, fmt.Errorf("add 64: %w", err)


Same, perhaps "overflow" or " Add64" would be better.

Fixed in 817ff4d.

potuz · 2026-05-26T13:37:37Z

 	require.NoError(t, err)
 	_, _, err = InitializePrecomputeValidators(t.Context(), s)
-	require.ErrorContains(t, "could not read every validator: addition overflows", err)
+	require.ErrorContains(t, "add 64: addition overflows", err)


interesting, this one actually added the "overflow" statement.

Fixed in 817ff4d.

potuz · 2026-05-26T13:48:22Z

-			Err:    errors.Wrap(err, "could not determine exited validator indices"),
-			Reason: Internal,
+	var (
+		activatedIndices, exitedIndices, slashedIndices, ejectedIndices []primitives.ValidatorIndex


An only nit is that these variables are not initialized so if no val is activated for example, this function will return nil when before it would return an initialized empty slice. This is fine as far as I can see in the current code.

Fixed in 6039980.

potuz · 2026-05-26T13:55:40Z

+	slashingsVector := params.BeaconConfig().EpochsPerSlashingsVector
+
+	for idx, validator := range requestedState.ValidatorsReadOnlySeq() {
+		if validator.ActivationEpoch() == requestedEpoch {


This fixes the linked issue, but the test TestActivatedValidatorIndices was removed, it would be nice to add a regression unit test that actually tests without a request epoch being zero so that we can assert that the indices were not in the reply (the previous test covered this case but asserted the opposite).

Fixed in 6039980.

james-prysm · 2026-05-26T14:41:39Z

+
+		for i := range b.validatorsMultiValue.Len(b) {
+			v, err := b.validatorsMultiValue.At(b, uint64(i))
+			if err != nil {


do we not need to log errors for this stuff? or does that just clutter?

This case is basically impossible, since the only way At returns on error is in case we have an out of bound error.

At is between 0 and Len(b), and we have a lock at the start of the function so no mutation of validatorsMultiValue is possible ==> No out of bound error can happen.

Added a log however in 3245eb5.

james-prysm

LGTM

nalepae force-pushed the func-iterator branch from f946caa to 3c40bdc Compare May 25, 2026 11:25

ActivatedValidatorIndices: Fix #16842

36296ba

Previous implementation was returning ALL active validators, contrary to what is stated by the godoc.

nalepae force-pushed the func-iterator branch from 9f45ab1 to 263a946 Compare May 25, 2026 12:24

ReadFromEveryValidator: Use a primitives.ValidatorIndex instead o…

c7032cc

…f an `int`. Rationale: Most of callers had to do this cast. Now the cast is directly done in the function, which is simpler for callers.

nalepae force-pushed the func-iterator branch from 263a946 to 0f18b06 Compare May 25, 2026 12:26

nalepae changed the title ~~ReadFromEveryValidator==> AllValidatorsReadOnly(func iterator)~~ Optimize beacon API (REST and gRPC) regarding endpoints needing to iterate over the validator set. May 25, 2026

nalepae force-pushed the func-iterator branch from 0f18b06 to 606f593 Compare May 25, 2026 12:35

nalepae changed the title ~~Optimize beacon API (REST and gRPC) regarding endpoints needing to iterate over the validator set.~~ Optimize beacon API (REST and gRPC) regarding endpoints needing to iterate over the whole validator set. May 25, 2026

nalepae added 7 commits May 25, 2026 15:39

ReadFromEveryValidator ==> ValidatorsReadOnlySeq (func iterator).

47b997c

This (very slightly) adds some complexity to the implementation of `ValidatorsReadOnlySeq` (which is done only once), but it simplifies the call sites (which are numerous).

/prysm/v1/validators/head/active_set_changes: Avoid up to 2.3M vali…

7fd56ea

…dators (mainnet) heap allocation by using `requestedState.ValidatorsReadOnlySeq` instead of `requestedState.Validators`.

/eth/v1/beacon/states/{state_id}/validator_count: Take an iterator …

8d3891d

…over validators as an input instead of the full validators slice. It avoids a ~2.3M validators (mainnet) materialization.

/eth/v1/beacon/states/{state_id}/validator_identities (SSZ response…

9088e4d

…): Avoid full validators list (2.3M on mainnet) materialization.

/eth/v1/beacon/states/{state_id}/validator_identities (JSON respons…

9550c62

…e): Avoid the full validators list (~2.3M on mainnet) materialization.

/eth/v1/beacon/states/{state_id}/validators: Avoid full validators …

e9eb314

…list materialization.

Add changelog.

77eb397

nalepae force-pushed the func-iterator branch from 606f593 to 77eb397 Compare May 25, 2026 13:40

nalepae marked this pull request as ready for review May 25, 2026 14:09

potuz reviewed May 26, 2026

View reviewed changes

Fix @potuz add64 comment.

817ff4d

james-prysm reviewed May 26, 2026

View reviewed changes

nalepae added 3 commits May 26, 2026 16:59

Fix @potuz comment by adding tests on ValidatorActiveSetChanges.

6039980

Fix @potuz comment: Replace nil by emply slice in response.

a41ae5b

Fix @james-prysm's comment: Add error log.

3245eb5

james-prysm approved these changes May 26, 2026

View reviewed changes

nalepae added this pull request to the merge queue May 27, 2026

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks May 27, 2026

nalepae added this pull request to the merge queue May 27, 2026

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks May 27, 2026

nalepae added this pull request to the merge queue May 27, 2026

Merged via the queue into develop with commit 4731880 May 27, 2026
23 checks passed

nalepae deleted the func-iterator branch May 27, 2026 12:11

Uh oh!

Conversation

nalepae commented May 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

potuz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nalepae May 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

james-prysm left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

nalepae commented May 24, 2026 •

edited

Loading

nalepae May 26, 2026 •

edited

Loading