Skip to content

Improve upstream resilience: add probe mode and remove force-revive#5603

Merged
vstakhov merged 2 commits into
masterfrom
vstakhov-upstream-revive-rework
Sep 4, 2025
Merged

Improve upstream resilience: add probe mode and remove force-revive#5603
vstakhov merged 2 commits into
masterfrom
vstakhov-upstream-revive-rework

Conversation

@vstakhov

@vstakhov vstakhov commented Sep 4, 2025

Copy link
Copy Markdown
Member

Improve upstream resilience: add probe mode and remove force revival

  • Problem: when all upstreams are marked down, Rspamd immediately revived all of them, causing tight retry loops (especially with 1–2 upstreams) and hammering bad endpoints.
  • Solution: implement probe mode with half-open checks and exponential backoff with jitter; remove force-revive-on-empty behavior. When no upstreams are alive, a single probe is allowed once its next_probe_at is due; on success, the upstream is reactivated; on failure, backoff increases.

Key changes

  • src/libutil/upstream.c
    • Add probe state per upstream: next_probe_at, probe_backoff, half_open_inflight.
    • Change rspamd_upstream_get_common to select a single probe candidate when no alive upstreams; otherwise return NULL.
    • Update rspamd_upstream_fail/rspamd_upstream_ok to manage probe backoff and activation.
    • Initialize probe scheduling in rspamd_upstream_set_inactive.
  • src/libserver/cfg_file.h
    • Add config knobs: upstream_probe_max_backoff, upstream_probe_jitter.
  • rspamd_upstreams_library_config
    • Wire new config options into limits.

Config

  • upstream_probe_max_backoff (default: 600s)
  • upstream_probe_jitter (default: 0.3)

Compatibility

  • Upstream selection may return NULL when all upstreams are down and no probe is due; existing call sites largely handle this. Added explicit NULL handling in fuzzy_backend_redis.c paths.

Impact

  • Prevents flapping and thundering herd on unhealthy endpoints.
  • Safer behavior for small upstream pools; controlled recovery via probes.

@vstakhov
vstakhov merged commit befcaf3 into master Sep 4, 2025
27 checks passed
@vstakhov
vstakhov deleted the vstakhov-upstream-revive-rework branch September 4, 2025 20:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant