The code is cheap and only meant to demonstrate the feasibility, but the concept ack revolves around these points imo:

• We can play dice with share-received events to simulate running the vardiff algorithms over arbitrary ranges of time. But we need to mock SystemTime::now() and add a way to bulk add new shares.
• With fake time simulations we can do large scale vardiff trials of whatever metrics we want and contrast against correlated attributes like target shares-per-minute.
  • I was interested in convergence time, stable-state jitter, and convergence accuracy
  • But responsiveness to external change is also a key capability, (how fast to adjust to a 50% spike/dip in hashrate)
• With this compilation of reproducible test results compiled into a profile we can use integration tests to lock in established performance thresholds and ratchet up the expectations if we find better algorithms.

after some optimization I got
https://github.com/adammwest/stratum/blob/feat/vardiff_kalman/sv2/channels-sv2/sim/vardiff_best.md
with the Bayesian model

after some optimization I got https://github.com/adammwest/stratum/blob/feat/vardiff_kalman/sv2/channels-sv2/sim/vardiff_best.md with the Bayesian model
Method Result
Bayesian model Good, Best result
Kalman filters Good
Jurik moving averages Good but artifacts on hashrate changes
Thompson sampling Bad

I'm so excited to see people other people nerding out on vardiff with me! Thank you!

A couple things I noticed in your results, the 2m convergence time is impressive but your response to a 50% hashrate drop only succeeds in readjusting 4.4% of the time. I'm not sure how best to balance those two metrics but probably not one at the expense of the other.

Some learning's I had @gimballock

• This task is hard, I think the current implementation is optimized to a degree.
• the most critical thing is the fitness, currently there are many metrics, how you combine all of them into a final value is what determines the goodness for any algorithm, as its a summary there are ways to game it.
• Use every value in the toml file, if you dont those values will naturally will degrade.
• There are many ways to combine many numbers which lead to slightly better and slightly worse performance.
  one thing I did which helped was to separate fitness into 2 categories improvement and regression, even better separating these per group e.g stable,coldstart ,... then you can decompose the value.
• Normalize each metric/group otherwise the more numerous or larger numbers will be the focus.
• There are many cases where you can get a good score, but the fitness prefers optimizing 1 variable or a set of variables at the expense of others.
• For grid parameter sweeps, they usually discretize the domain so you are bounded in improvement only by dimension range and amount of queries. so you need to constantly increase queries or shrink ranges. usually you are limited due to time. For this reason I prefer random restart hill climbing I find is generally pretty good when you don't make assumptions about the data.
• If you have too many parameters to optimize you can over fit, and end up just gaming the test

Thanks for these insights @adammwest — especially the point about fitness decomposition and normalization. A lot of what you're describing matches the evolution I've gone through on this PR, so let me give a timeline of
how the approach has matured:

Phase 1: Basic metrics + simulation harness

Initially I focused on three metrics I thought were important: convergence time, jitter, and accuracy. These were evaluated via a time-compressed simulation that replays a synthetic share stream through the vardiff
algorithm. This gave us reproducible, large-scale trials (50 cells × 1000 trials) against correlated attributes like target shares-per-minute.

Phase 2: Decomposed pipeline model

I wanted to make algorithm search more systematic, so I decomposed "a vardiff algorithm" into four independent, replaceable components: estimator, statistic, boundary, and decision rule. The idea was to mix-and-match
implementations at each slot for the best composite.

This model worked well for the classic algorithm, the parametric variant, and the EWMA approach. But when I tried to embed a Bayesian model, it broke down — the components aren't truly independent. There's a sequential
data flow: the estimator needs to communicate its belief to the boundary ("should we respond?") and to the update rule. Additionally, since vardiff triggers on a timer rather than on share arrival, the decision rule needs
to call back to the estimator to update state when adjustments occur. I also dropped the "statistic" component as it wasn't pulling its weight.

The resulting three-stage pipeline (Estimator → Boundary → UpdateRule) is what's in this PR. It successfully hosts the classic algorithm, EWMA, AdaCUSUM, and could host a Bayesian approach.

Phase 3: Aggregate fitness metric

To your point about "how you combine all metrics into a final value" — we now have a configurable aggregate metric that allows weighting across the underlying measurements. This addresses exactly the gaming concern you
raised: rather than optimizing one metric at the expense of others, we can define a weighted composite that represents our desired tradeoff. The regression baseline locks in the full vector of metrics so we catch
regressions in any dimension, not just the aggregate.

Your suggestion to separate fitness into improvement vs. regression categories per scenario group (stable, coldstart, reaction) is a good one. Currently the regression test does compare per-cell, so a coldstart regression
can't hide behind a stable-state improvement, but making this more explicit in the scoring would help.

Phase 4: Realistic operating conditions

After discussions with hardware engineers, I retuned the test scenarios to realistic share rates (2–30 spm instead of the earlier 6–120 range). The engineers confirmed that responding to partial hardware failures and
network slowdowns on an established channel is valuable functionality — even though in practice many hashrate changes currently cause miner reconnections (which resets vardiff anyway). We've been doing live testing with
physical miners on testnet4 and confirmed this pattern: when vardiff ramps difficulty too aggressively, it can interact with firmware timeout behaviors in ways that force reconnections, making reactivity testing harder
than expected.

Current direction

I've backed off from prioritizing convergence speed after seeing overcorrection in practice. The current focus is on:

• Stability under steady-state (minimal oscillation once converged)
• Reasonable reactivity (detect genuine changes within 2–3 retarget windows, not 1)
• Asymmetric cost awareness — difficulty increases are more disruptive than decreases. An overshoot upward causes difficulty-too-low share rejections (wasted miner work), while an undershoot downward just means slightly
  more shares than optimal (cheap). The AsymmetricCusumBoundary encodes this: it requires stronger evidence before raising difficulty than lowering it. We can now actually measure the impact via the share-rejection metrics
  (shares_rejected_total{reason="difficulty-too-low"}) that were recently added to the pool's monitoring (sv2-apps PR Docs: Channel Factory #491).

On your point about normalization: agreed, and the per-metric tolerance budgets in the regression test (absolute slack + optional multiplicative slack) are our current mechanism for this. Open to suggestions on better
normalization approaches

I've been working on a simple proxy to validate the vardiff responsiveness findings from the simulations. My first test aimed to confirm the assessment that the existing vardiff algorithm is slow to respond to hashrate changes after it has already converged.

Test Setup:

I configured an S21 miner → tproxy (vardiff disabled) → shape-proxy (controllable share rate) → SRI pool. The shape-proxy maintains a smoothed share rate based on current pool difficulty and can selectively drop shares to simulate hashrate changes on command.

Methodology:

I ran two parallel instances of the unmodified SRI pool server (main branch) and triggered a 50% share rate drop via API command. I then measured how long it took for the pool's hashrate estimate and share acceptance rate to adjust to the new rate.

Results:

The pool required approximately 40 minutes to complete the initial hashrate adjustment, while the simulation predicted 70% of miners failed to adjust at all and of those that did it took 5m. So it may look like a 20× discrepancy between simulation and observed behavior, its more of a confirmation that it takes a very long time.

I will investigate if there is any discrepancy and re-calibrate the simulation if so. The attached image shows the resulting hashrate and accepted share rate for both trials.

The blue lines are the pool hashrate, you see the initial spike after it first starts up is the convergence spike then is settles and stays flat for not-quite an hour before I activate the 50% hashrate drop, immediately visible in the share rate drop. This stays flat until eventually vardiff fires the adjustment allowing the share rate to return to normal.

Hi, this is quite a detailed analysis and great decomposition of relevant parts.

Have you had a look at how ckpool implements this? I think he has quite naturally arrived at a very optimal state, balancing the different metrics. This is the repo, you'll have to grep through to find the vardiff implementation: https://github.com/ckolivas/ckpool

I've re-implemented his approach in Rust as well and made it a bit more configurable here: https://github.com/parasitepool/para/blob/master/src/vardiff.rs

Would be interesting to see how that algorithm performs in your benchmarks.

Hi, this is quite a detailed analysis and great decomposition of relevant parts.

Have you had a look at how ckpool implements this? I think he has quite naturally arrived at a very optimal state, balancing the different metrics. This is the repo, you'll have to grep through to find the vardiff implementation: https://github.com/ckolivas/ckpool

I've re-implemented his approach in Rust as well and made it a bit more configurable here: https://github.com/parasitepool/para/blob/master/src/vardiff.rs

Would be interesting to see how that algorithm performs in your benchmarks.

Thanks for the info @paratoxicdev , I will add this to my investigation. I know that is a sv1 native pool but i will see what bits of his research crossover to sv2 context and see if it's competitive!

Here is a breakdown of the calibration comparisons I made between the real hashrate tests and the simulation results confirming the predictions with the understanding that; with this algorithm responsiveness scales with the age of the connection. Suggesting that this detail be included in the simulation so metrics are more directly comparable:

⏺ Vardiff Calibration Summary: Simulation vs Real Miner Testing

  Algorithm under test: Classic Parametric vardiff (VardiffState from stratum-core, branch vardiff/parametric-thresholds)

  Hardware: Antminer S21 (~200 TH/s), testnet4
  Pool config: shares_per_minute = 6 and shares_per_minute = 20
  Tool: shape-proxy with Step{1.0→0.5, at_secs:300} and Track{1.0} reset

  ---
  Structural findings confirmed
  
  ┌────────────────────────────┬───────────────────────────────────────────────────────┬────────────────────────────────────────────────────────┬───────┐
  │          Property          │                    Sim prediction                     │                    Real observation                    │ Match │
  ├────────────────────────────┼───────────────────────────────────────────────────────┼────────────────────────────────────────────────────────┼───────┤
  │ Steady-state jitter        │ 0.000 fires/min                                       │ Zero fires during 30+ min stable operation (both       │ Yes   │
  │                            │                                                       │ rates)                                                 │       │
  ├────────────────────────────┼───────────────────────────────────────────────────────┼────────────────────────────────────────────────────────┼───────┤
  │ Per-fire step magnitude    │ Deterministic: realized_spm / target_spm ratio        │ -16.7% consistently (5 consecutive fires)              │ Yes   │
  ├────────────────────────────┼───────────────────────────────────────────────────────┼────────────────────────────────────────────────────────┼───────┤
  │ Fire cadence after counter │ Exactly 300s (15% threshold at delta_time≥300)        │ 22:39→22:44→22:49→22:54→22:59 (5-min cadence)          │ Yes   │
  │  reset                     │                                                       │                                                        │       │
  ├────────────────────────────┼───────────────────────────────────────────────────────┼────────────────────────────────────────────────────────┼───────┤
  │ Overshoot after staircase  │ p99 = 69% at 6 spm                                    │ ~60% overshoot → share flood → oscillation             │ Yes   │
  │ descent                    │                                                       │                                                        │       │
  ├────────────────────────────┼───────────────────────────────────────────────────────┼────────────────────────────────────────────────────────┼───────┤
  │ Sensitivity depends on     │ Implicit in algorithm (accumulating counter never     │ 5-min counter: 4.4 min reaction; 51-min counter: 51.8  │ Yes   │
  │ counter age                │ resets except on fire)                                │ min reaction                                           │       │
  ├────────────────────────────┼───────────────────────────────────────────────────────┼────────────────────────────────────────────────────────┼───────┤
  │ Algorithm symmetric for    │ Sim shows similar reaction rates both directions      │ Confirmed: similar timescales for step-down and        │ Yes   │
  │ ±50%                       │                                                       │ step-up                                                │       │
  └────────────────────────────┴───────────────────────────────────────────────────────┴────────────────────────────────────────────────────────┴───────┘

  ---
  Reaction time: -50% step (hashrate halved)
  
  ┌────────────────────────────────────┬──────────────────────────────────────────────────────────┬──────────────────────────────────┐
  │             Condition              │                      Sim prediction                      │           Real result            │
  ├────────────────────────────────────┼──────────────────────────────────────────────────────────┼──────────────────────────────────┤
  │ 6 spm, fresh counter (~5 min)      │ 3.7% fire within 5 min; those that fire: p50=5m          │ Slot 3: 4.4 min, Slot 4: 8.0 min │
  ├────────────────────────────────────┼──────────────────────────────────────────────────────────┼──────────────────────────────────┤
  │ 6 spm, settled counter (~51 min)   │ ~96% don't fire within 5 min (tail extends indefinitely) │ Slot 3: 51.8 min                 │
  ├────────────────────────────────────┼──────────────────────────────────────────────────────────┼──────────────────────────────────┤
  │ 20 spm, moderate counter (~27 min) │ 14.2% fire within 5 min                                  │ Slot 3: 6.9 min, Slot 4: 8.4 min │
  └────────────────────────────────────┴──────────────────────────────────────────────────────────┴──────────────────────────────────┘

  ---
  Reaction time: +50% step (return to full rate)
  
  ┌───────────────────────────┬──────────────────────────────────┬──────────────────────────────────┐
  │         Condition         │          Sim prediction          │           Real result            │
  ├───────────────────────────┼──────────────────────────────────┼──────────────────────────────────┤
  │ 6 spm, 51-min counter     │ Deep in tail (>96% non-reactive) │ Slot 3: 51.8 min                 │
  ├───────────────────────────┼──────────────────────────────────┼──────────────────────────────────┤
  │ 6 spm, 68-min counter     │ Deep in tail                     │ Slot 4: 15.4 min                 │
  ├───────────────────────────┼──────────────────────────────────┼──────────────────────────────────┤
  │ 20 spm, 20-21 min counter │ Higher spm improves detection    │ Slot 3: 5.9 min, Slot 4: 6.3 min │
  └───────────────────────────┴──────────────────────────────────┴──────────────────────────────────┘

  ---
  Key insight: counter age is the dominant variable
  
  The sim's test design (step at t=15min, 5 min after cold-start convergence) always tests with a fresh counter. This produces the "3.7% react within 5 min"
  figure. In reality, a pool that hasn't fired in hours has a massive accumulated counter that dilutes any step signal — explaining 40-minute to 2.5-hour
  response times observed in earlier tests.

  At 20 spm vs 6 spm with the same counter age (~20 min), reaction time improves ~8x (51.8 min → 5.9 min). This is because more shares per minute means the
  new rate accumulates statistical weight faster against the counter history.

  ---
  Simulation validity assessment
  
  The simulation is trustworthy for relative algorithm comparisons. It correctly models:
  - The threshold table mechanics (fire/no-fire decisions)
  - The deterministic staircase behavior post-fire
  - The zero-jitter steady state
  - The share-rate dependence on detection speed
  - The fundamental sensitivity-decay-over-time flaw

  Gap to address: The sim should add a "counter age" axis (test steps at t=30m, t=60m, t=120m) to characterize the tail distribution, which is where
  real-world pools spend most of their time. The current 5-minute observation window and 15-minute step timing understate the algorithm's poor real-world
  responsiveness.```  

Ok here is evidence that the top algorithm (deployed side-by-side with the current vardiff) is immune to the age-dependence effect. This reproduces results predicted in the simulation but now seen in real life.

I started a mining channel against both pools and let them mine overnight.
In the morning I dropped the hashrate in half for both pools at roughly the same time and waited till both pool's vardiffs finished adjusting to the new hashrate changes. The annotated image below shows the results from the grafana dashboard.

The current vardiff algorithm not only took several hours to respond, the response it eventually made was in the wrong direction! While the new algorithm responded in a few minutes and settled on the correct value.