Momentum samplers for flow models (EMA momentum, damped extrapolation, any-sampler filter)
Momentum on the data prediction for DPM-Solver++-style samplers on flow-matching models: a negative coefficient lowers the error at equal step count, a positive one only adds lag. Nodes: Sampler Momentum, Sampler Momentum Filter (wraps any one-evaluation sampler, unevaluated), Sigmas grid; KSampler entries ddim_momentum, dpmpp_2m_sde_damped and <base_momentumized. Includes a diagnosis of why the 2023 dpmpp_dualsde_momentumized fails on flow models.
Momentum in diffusion samplers: what it is, why dpmpp_dualsde_momentumized fails, and what works
Languages: English · Русский · 中文 · Марий
This English file is the original and the authoritative version. The Russian and Chinese files are full translations of it; the Meadow Mari file is a full machine-assisted translation that has not been reviewed by a native speaker. Where a translation differs from this file, this file is right.
Preprint: 10.5281/zenodo.24214606. Code: YakovSava/momentum-cfg-samplers-research.
Status (2026-11-14): finished; every section below is filled in. Hypotheses H1-H9 were registered (Section 3) before any measurement; sections that are post hoc are marked as such.
Short answer
- Why the shipped sampler produces garbage. Not because of momentum (through
KSamplerits coefficient is 0).dpmpp_dualsde_momentumizedis written for variance-exploding models: its noise step has the wrong variance on a flow-matching model such as Anima, so witheta = 1the latent blows up (std 3-4.5 instead of 1.2). Witheta = 0the same code draws a cat. Two further defects: a stacked 3M correction that is counted on top of an already second-order step (the sampler diverges even without noise, 2-35x worse thandpmpp_2m), and a parameterrthat is overwritten inside the loop (Sections 1.3, 4). - What "momentum" has to be. The shipped one is an exponential moving average with a positive coefficient, i.e. a low-pass filter on the model output: it adds lag and increases the error with the coefficient (up to +61%). With a negative coefficient the same recursion is a stable extrapolation and lowers the error (Section 5).
- Gain at one evaluation per step, equal NFE. Deterministic:
ema_timedm = -0.5 /emam = -0.6, 11-21% lower error thandpmpp_2m, about 1.2x fewer steps for the same error, better LPIPS in 14/15 units at 8 steps. Stochastic: damped extrapolation (beta= 0.5) oremam = -0.3, 6-15% lower strong error thandpmpp_2m_sde; strong negative momentum hurts with noise. One-evaluation methods beat the two-evaluationdpmpp_sdeat equal NFE by 16-29%. - This is a modest speed-up, not a factor of two. A static step grid from the previous study cut the
error by half on the prompts it was derived on, but it did not transfer to the held-out
catprompt or to CFG 8 / 1216x832 (Sections 6, 7); it is shipped as an experimental node only. - Nodes:
Sampler Momentum,Sigmas (turning-angle grid)and the KSampler entriesddim_momentum,dpmpp_2m_sde_dampedand 19*_momentumized(Sections 10, 13).
Prior work (Heavy Ball momentum for diffusion solvers, the ComfyUI sonar and Seren Sampler packs and
others) is discussed in Section 14; momentum in diffusion sampling is not a new idea, and what this record
adds is stated there.
Follow-up to flow_zero_cost_proxy, flow-sde-research, flow-wls-research and
adaptive-step-samplers. Same conventions:
equal number of network evaluations (NFE), paired comparisons with bootstrap intervals over units,
negative results reported as such, README as the full record.
Question
The *_momentumized samplers of ComfyUI-Extra-Samplers (dpmpp_dualsde_momentumized,
clyb_4m_sde_momentumized, res_momentumized) are advertised as "momentum" variants of
DPM-Solver++. On Anima (flow matching, bf16, CFG 8, 30 steps) dpmpp_dualsde_momentumized
returns a washed-out silhouette instead of an image (figures/diagnosis_*.jpg), while
dpmpp_2m_sde_gpu returns a clean picture at the same settings.
Two questions:
- Diagnosis. Which component of that sampler destroys the image?
- Construction. Is there a form of "momentum" that, at one network evaluation per step and equal NFE, lowers the error against the converged solution, and if so with what coefficient?
1. Framework
1.1 Setting
ComfyUI hands the sampler the denoised estimate D(x, sigma). For a flow-matching model
x = alpha * x0 + sigma * eps with alpha = 1 - sigma; half-logSNR lambda = log(alpha / sigma).
For a variance-exploding model alpha = 1, lambda = -log sigma. Over one step from sigma_s to
sigma_t (h = lambda_t - lambda_s > 0, eta >= 0, h_eta = (1 + eta) h) the exponential
integrator in the data-prediction form (DPM-Solver++ SDE, as in ComfyUI core) is
x_t = (sigma_t / sigma_s) exp(-eta h) x_s
+ alpha_t (1 - exp(-h_eta)) D_eff
+ sigma_t sqrt(1 - exp(-2 eta h)) z, z ~ N(0, I). (1)
With D_eff = D_s this is first order (DDIM for eta = 0). Every method studied here differs only
in D_eff, built from the evaluations the method already has:
| name | D_eff | parameter | note |
| --- | --- | --- | --- |
| extrapolate | D_s + beta * c * (D_s - D_prev) / r, r = h_prev / h, c = 1/2 | beta | beta = 1 is DPM-Solver++(2M) SDE midpoint, exactly ComfyUI's dpmpp_2m_sde; beta = 0 is DDIM / first order |
| ema | Dbar_s = m * Dbar_prev + (1 - m) * D_s | m | the Clybius "momentum" on the data term; m > 0 low-pass (lag), m < 0 recursive extrapolation |
| ema_timed | as ema with m_s = m (sigma_s / sigma_max - m / 2) | m | the exact time scaling of ComfyUI-Extra-Samplers |
| pec | predictor extrapolate(beta = 1) to x~_t, evaluate D~_t = D(x~_t, sigma_t), corrector D_eff = (1 - gamma) D_s + gamma D~_t; D~_t is reused as D of the next step | gamma | Adams predict-evaluate-correct, 1 NFE per step; gamma = 1/2 is the trapezoid (Heun-type) corrector; the ODE analogue of Nesterov's look-ahead |
All four cost one evaluation per step, so a budget of N evaluations is N grid points of the
normal scheduler for all of them. The noise z in pec is the same in predictor and corrector.
1.2 Proposition 1 (heavy ball on x is inconsistent; on increments it is Adams-Bashforth)
Polyak's heavy ball x_{n+1} = x_n + h v_n + mu (x_n - x_{n-1}) applied to dx/dt = v has
increment weights summing to h + mu h_{n-1}, not h: it is not a consistent integrator for
mu != 0 (it integrates a rescaled time). The consistent version normalises the weights,
x_{n+1} = x_n + (1 - mu) h v_n + mu (h / h_{n-1}) (x_n - x_{n-1}), which is an exponential moving
average of the increments; for mu = -1/2 and uniform steps it is Adams-Bashforth 2. So
"momentum" on increments with a positive coefficient is a lag filter (order 1 with a larger constant
than Euler), and with a negative coefficient it is a multistep extrapolation. In the exponential
integrator frame the increment is driven by D, which gives the ema and extrapolate rows above.
The *_momentumized samplers use mu > 0 by default (0.5 in the node, 0 through KSampler).
1.3 Proposition 2 (what dpmpp_dualsde_momentumized computes)
Per step it takes a full DPM-Solver++(2S) SDE step (two evaluations, second order, D_eff from the
midpoint evaluation), and then adds the multistep correction of DPM-Solver++(3M) SDE
(phi_2 d1 - phi_3 d2 built from the last three D values) on top of the result. The 2S step already
contains the second-order term, so the correction is counted twice. Each step therefore carries a
spurious O(h^2) term of the same size as the second-order correction itself; over N steps this is
an O(h) global bias, not a convergent method. In addition the sampler uses the variance-exploding
formulas (t = -log sigma, noise sigma_up z without alpha factors) on a flow model, for which
ComfyUI core has switched to (1). Through KSampler the momentum coefficient is 0, so the picture in
figures/ is produced without any momentum.
1.4 Evaluation
- Model: Anima (
anima_aestheticV11, bf16), 576x768, CFG 5,normalscheduler (shift 3), 16-channel latent. 5 prompt classes (flat,face,fur,grid,cat;catis the prompt of the failing picture) x 3 seeds (42, 1337, 7) = 15 units. - ODE reference (
eta = 0): Adams-Bashforth 2 on 192 uniform points oft(the 12 references offlow_zero_cost_proxyare reused unchanged, the 3catunits are added). Its own error is of the order of 5%. - SDE reference (
eta = 1):dpmpp_2m_sdeof ComfyUI core at 192normalsteps with the Brownian-tree noise sampler seeded by the unit seed. Every method atNsteps uses the same tree, so they integrate the same Brownian path and the strong error is well defined. - Metric: relative L2 error of the final latent against the reference. LPIPS against the decoded
reference for the winners at
N = 8. - Budgets:
N = 8, 12, 16, 24evaluations. The two-evaluationdpmpp_sde(2S) of core is run atN/2steps as the equal-NFE baseline. - Statistics: paired log-ratios over the 15 units, 95% bootstrap interval (4000 resamples).
Percent =
exp(log-ratio) - 1. Effects below about 5% are not detectable. - Diagnosis run: the exact settings of the failing picture (1216x832, CFG 8, 30 steps, seed
1109260825298231,
normal), each ablation decoded tofigures/, plus the same ablations on the 15 units at 576x768 with the error against the SDE reference.
2. Code map
momentum.py (the four D_eff rules in frame (1); pure torch), wrap.py (the same filter as a model
wrapper for other samplers, Section 13), legacy.py (faithful port of dpmpp_dualsde_momentum with switches
for the ablation), schedules.py and anima_grids.json (sigma grids), toy.py (Gaussian toy flow for the
tests), rig.py, units.py, decode.py, run_reference.py, run_diagnosis.py, run_sweep.py,
run_grids.py, run_loo.py, export_loo_grids.py, run_user_check.py, perceptual_check.py,
analyze.py, analyze_grids.py, analyze_loo.py, __init__.py (ComfyUI nodes and KSampler entries),
tests/test_samplers.py (13 CPU tests).
3. Hypotheses (registered before any measurement)
Thresholds use the paired log-ratio with its 95% interval over the 15 units; "beats" means the interval excludes zero.
| # | Hypothesis | Supported if | Verdict |
| --- | --- | --- | --- |
| H1 | The failure of dpmpp_dualsde_momentumized is the stacked 3M correction (Prop. 2), not momentum and not the noise type | removing the correction (everything else unchanged, momentum = 0) lowers the error against the SDE reference by at least 50% (log-ratio <= -0.69) in every unit and brings it within 20% of core dpmpp_sde at the same NFE; momentum 0 vs 0.5 changes the error by less than 20% | refuted (Sections 4, 9): removing the correction lowers SDE error by 60-97% but the result stays 1.9-3.4x worse than core dpmpp_sde; at N = 24 one unit improves by only 9% |
| H2 | The variance-exploding noise form on a flow model is a bias, not the catastrophe | the port without the correction but with the old noise form is within x1.5 of core dpmpp_sde at every N | refuted (the VE noise form is the catastrophe, ratio to core dpmpp_sde 1.9-3.4 at every N) |
| H3 | Positive momentum (ema, m > 0) only hurts | m = 0.3 and m = 0.6 lose against m = 0 (= extrapolate with beta = 0) at every N, ODE and SDE, and the loss grows with m | supported (ODE and SDE, every N, intervals above zero, loss grows with m) |
| H4 | Damped extrapolation helps at small budgets | ODE: at N = 8 the best beta is in [0.5, 0.9] and beats beta = 1 (2M) by at least 10% (log-ratio <= -0.105); at N = 24 no beta in {0.75, 1} beats the other | refuted (best beta at N = 8 is 0, not in [0.5, 0.9]; beta = 0.75 beats 1 by only 3% at N = 8) |
| H5 | The SDE wants more damping than the ODE | at N = 8 and N = 12 the best beta for eta = 1 is at least 0.25 below the best beta for eta = 0 | refuted (best SDE beta is 0, 0, 0, 0.5, ODE 0, 0.25, 0.25, 0.75: the SDE prefers about the same damping) |
| H6 | Predict-evaluate-correct beats pure extrapolation at small budgets | ODE: pec with gamma = 1/2 beats extrapolate(beta = 1) by at least 10% at N = 8 and N = 12; the interval includes zero at N = 24 | refuted (pec beats 2M by 5% at N = 8 and 12, not 10%; it helps most at N = 24, -18%) |
| H7 | Time-scaled momentum (Clybius form) is no better than the best constant coefficient | ema_timed(m = 0.5) does not beat the best constant ema or extrapolate in any cell by more than 5% | supported for the registered positive m (never better than the best constant); the negative-m time scaling is a post hoc finding (Section 5) |
| H8 | The latent-L2 winner at N = 8 is also the perceptual winner | the winner's LPIPS to the decoded reference is below that of extrapolate(beta = 1) in at least 10 of 15 units at N = 8 | supported (LPIPS wins 14/15 at N = 8) |
| H9 | One-evaluation momentum beats the two-evaluation sampler at equal NFE | the best one-evaluation method beats core dpmpp_sde at N/2 steps at every N for eta = 1 | supported (best one-evaluation SDE beats core dpmpp_sde at N/2 steps by 16-29%) |
Sweep grid: ODE extrapolate beta in {0, 0.25, 0.5, 0.75, 1, 1.25}, ema m in {0.6, 0.3, -0.3, -0.6},
pec gamma in {0.25, 0.5, 0.75}, ema_timed m = 0.5; SDE extrapolate same set, ema
m in {0.6, 0.3, -0.3}, pec gamma = 0.5, plus core dpmpp_sde at N/2. Nothing outside this
grid is used for the verdicts; anything added later is marked post hoc.
4. Diagnosis of dpmpp_dualsde_momentumized (single run, user settings)
Settings: Anima, 1216x832, CFG 8, 30 steps normal, seed 1109260825298231, prompt cat. Error is the
relative L2 distance of the final latent to the 96-step ODE solution (extrapolate, eta = 0); std is
the standard deviation of the final latent (a clean latent has about 1.2). Stochastic samplers
integrate a different path than the ODE, so their error against an ODE reference is not small even
when they are correct (core samplers: 0.80-0.90); what matters is the gap to those and the std.
Picture: figures/diagnosis_sheet.jpg; numbers: results/diagnosis.json.
| variant | eta | what it is | error | std |
| --- | --- | --- | --- | --- |
| core dpmpp_2m_sde_gpu | 1 | reference behaviour | 0.90 | 1.47 |
| core dpmpp_sde_gpu | 1 | 2S SDE, flow-correct, 60 NFE | 0.80 | 1.54 |
| legacy_asis | 1 | the sampler as shipped, momentum = 0 | 2.77 | 3.18 |
| legacy_m05 | 1 | momentum = 0.5 | 2.20 | 3.06 |
| legacy_nocorr | 1 | stacked 3M correction removed | 3.33 | 4.30 |
| legacy_noshadow | 1 | r kept at 1/2 (no shadowing) | 4.21 | 4.50 |
| legacy_eta0 | 0 | shipped, no noise | 0.79 | 1.20 |
| legacy_eta0_nocorr | 0 | correction removed, no noise | 0.44 | 1.04 |
| legacy_eta0_noshadow | 0 | r kept at 1/2, no noise | 0.81 | 1.24 |
| extrapolate | 1 | new, beta = 1, 30 NFE | 0.70 | 1.37 |
| extrapolate | 0 | new, beta = 1, 30 NFE | 0.57 | 1.53 |
| pec gamma = 0.5 | 1 | new, 30 NFE | 0.79 | 1.51 |
Findings (one prompt, one seed; the 15-unit version is in Section 5):
legacy_nocorrreproduces the failing picture of the report (a pale blue silhouette);legacy_asisfails in a different way (orange noise); both have a latent std of 3-4.5. The failure is in the noise step, not in momentum: witheta = 0the same code produces a recognisable cat.- H1 is refuted. Removing the stacked correction does not repair the sampler (error 2.77 -> 3.33, std 3.18 -> 4.30). With the noise switched off it makes the result better (0.79 -> 0.44), so the double counting of Proposition 2 is real but secondary.
- H2 is refuted in the other direction. The variance-exploding noise form is not a mild bias on
Anima: it is the catastrophe. In the VE formulas the injected noise
sigma_upis computed forx = x0 + sigma * eps, while the flow model hasx = (1 - sigma) x0 + sigma * eps; the stationary variance is wrong and the error compounds over 30 steps. The evidence is indirect but tight:legacy_nocorr(variance-exploding 2S SDE) and coredpmpp_sde(flow-correct 2S SDE) are the same algorithm in different frames, and one gives a silhouette (error 3.3) while the other gives a clean picture (0.80); the 15-unit sweep gives 1.6-1.9 vs 0.6-0.85 (Section 9). The mechanism itself (the variance of the stationary distribution) was not isolated by a separate experiment. - The momentum coefficient is irrelevant to the failure: 0 and 0.5 are equally broken (2.77, 2.20).
- A second defect of the original, found while porting (the port is checked against the original to
1e-10intests/test_samplers.py): the 3M branch assignsr = h_1 / hto the function parameterr, so after the second step the midpoint of the 2S stage moves from1/2toh_1 / h. Witheta = 0this does not matter much (0.79 vs 0.81), with noise it changes the picture but does not rescue it. This was found after the hypotheses were registered and is not in the table of Section 3. - Flow-correct one-evaluation samplers (
extrapolate,pec) give clean pictures at the same settings.
5. ODE sweep (eta = 0, 15 units, equal NFE)
Full tables with intervals and win counts: results/sweep.md, raw numbers results/sweep_ode.json.
Mean relative error to the ODE reference; the log-ratio is paired against DPM-Solver++(2M)
(extrapolate_1.0, ComfyUI dpmpp_2m), 95% bootstrap interval, negative is better. Percent = exp(lr) - 1.
| variant | N = 8 | N = 12 | N = 16 | N = 24 |
| --- | --- | --- | --- | --- |
| 2M (extrapolate, beta = 1), error | 0.718 | 0.580 | 0.497 | 0.321 |
| extrapolate beta = 0 (first order) | -0.062 [-0.084, -0.042] | -0.043 [-0.086, +0.005] | -0.040 [-0.068, -0.013] | +0.053 [-0.015, +0.134] |
| extrapolate beta = 0.75 | -0.034 [-0.048, -0.021] | -0.034 [-0.054, -0.016] | -0.043 [-0.069, -0.019] | -0.032 [-0.070, -0.001] |
| ema m = +0.3 | -0.002 [-0.024, +0.026] | +0.053 [+0.006, +0.110] | +0.055 [+0.012, +0.099] | +0.253 [+0.186, +0.330] |
| ema m = +0.6 | +0.089 [+0.047, +0.139] | +0.240 [+0.157, +0.323] | +0.285 [+0.204, +0.374] | +0.478 [+0.361, +0.605] |
| ema_timed m = +0.5 (Clybius form) | +0.008 [-0.017, +0.037] | +0.088 [+0.042, +0.143] | +0.071 [+0.026, +0.115] | +0.288 [+0.218, +0.368] |
| ema m = -0.3 | -0.112 [-0.157, -0.072] | -0.099 [-0.136, -0.064] | -0.121 [-0.175, -0.074] | -0.092 [-0.166, -0.017] |
| ema m = -0.6 | -0.117 [-0.168, -0.072] | -0.108 [-0.162, -0.056] | -0.199 [-0.275, -0.126] | -0.232 [-0.312, -0.153] |
| pec gamma = 0.5 | -0.054 [-0.109, -0.010] | -0.044 [-0.082, -0.003] | -0.106 [-0.184, -0.034] | -0.196 [-0.293, -0.092] |
| shipped dualsde (eta = 0, 2 NFE per step, N/2 steps), error | 24.7 | 10.1 | 7.0 | 2.0 |
| same, stacked correction removed, error | 0.679 | 0.603 | 0.458 | 0.382 |
Post hoc extension (names ph_* in the raw file; chosen after seeing that negative m helps; these rows are
not part of the registered grid), same 15 units:
| variant | N = 8 | N = 12 | N = 16 | N = 24 |
| --- | --- | --- | --- | --- |
| ema m = -0.45 | -0.129 [-0.195, -0.075] | -0.103 [-0.145, -0.064] | -0.164 [-0.234, -0.097] | -0.191 [-0.257, -0.125] |
| ema m = -0.75 | -0.088 [-0.127, -0.051] | -0.112 [-0.170, -0.057] | -0.195 [-0.264, -0.126] | -0.189 [-0.280, -0.094] |
| ema m = -0.9 | -0.008 [-0.049, +0.044] | +0.004 [-0.058, +0.069] | -0.041 [-0.126, +0.057] | +0.051 [-0.090, +0.212] |
| ema m = -1.2 | +0.454 | +1.159 | +1.832 | +3.245 (diverges) |
| ema_timed m = -0.5 | -0.129 [-0.183, -0.081] | -0.116 [-0.175, -0.062] | -0.192 [-0.273, -0.114] | -0.241 [-0.323, -0.157] |
| ema_timed m = -0.8 | -0.065 [-0.114, -0.011] | -0.012 [-0.101, +0.086] | -0.086 [-0.178, +0.015] | -0.021 [-0.348, +0.293] |
| pec gamma = 0.4 | -0.040 [-0.067, -0.015] | -0.041 [-0.070, -0.010] | -0.092 [-0.145, -0.038] | -0.198 [-0.275, -0.117] |
| pec gamma = 0.6 | -0.050 [-0.105, -0.003] | -0.050 [-0.094, -0.000] | -0.104 [-0.209, -0.016] | -0.174 [-0.264, -0.081] |
What this says:
- The sign of the momentum matters, and the shipped default has the wrong sign. A positive
coefficient is a low-pass filter on the data prediction; it adds lag and hurts monotonically with the
coefficient (up to +61% error for
m = 0.6). A negative coefficient is a recursive extrapolation and helps:m = -0.6is 11% (N = 8) to 21% (N = 24) lower than 2M; log-ratio -0.117 ... -0.232, intervals exclude zero in every cell, wins 14/15 at N = 8 and 15/15 at N = 16. - Size of the gain in NFE. Interpolating the mean error curve in
log N, 2M needs about 1.2 times the budget to reach the error ofemam = -0.6(1.20, 1.23, 1.20 at N = 12, 16, 24). That is a real but modest saving: about 17% fewer evaluations at equal error, not a factor of two. - The coefficient has a stability edge. The recursion
Dbar = m Dbar_prev + (1 - m) Dis stable for|m| < 1; atm = -0.9the gain is gone and atm = -1.2it diverges (error 8 at N = 24). The best region is-0.75 <= m <= -0.45. The time-scaled formema_timedwithm = -0.5(coefficient-0.5 (t + 0.25), from -0.625 at the start to -0.125 at the end) matches the best constant at every N with one parameter. With a positivemthe same time scaling is among the worst variants (+33% at N = 24). - 2M itself is not the best one-evaluation member. Damped extrapolation (
beta0.25-0.75) beats it by 3-6% at N <= 16, and at N = 8 the first-order method (beta = 0) is better than 2M by 6%: with 8 steps and a large first step inlambdathe second-order term extrapolates noise. pec(predict, evaluate at the predicted point, correct; the evaluation is reused) is weaker thanemaat small N (-5%) and equal to it at N = 24 (-20%).- The shipped sampler diverges even without noise. With
eta = 0it is 35 times worse than 2M at N = 8 and still 6 times worse at N = 24 (2 NFE per step, so N / 2 steps). Without the stacked correction it is within 5-20% of 2M, so the correction is a real second defect (Proposition 2).
6. Does the angle-rate grid of flow_zero_cost_proxy stack with it?
That study found that a static grid with step density proportional to the mean angle rate, optionally
weighted by the measured error propagation (propagated), lowers Euler error by 30-50%. Here the same
grids are tried with the new samplers (run_grids.py, run_loo.py, analyze_loo.py,
results/grids_heldout.md). This section is post hoc. The stored grid (anima_grids.json) was fitted on
all 12 old units, so it is in sample for them. Two out-of-sample schemes are therefore reported: loo
(grid from the other 11 units: excludes the unit itself but not its two other seeds of the same prompt)
and lpo (leave-prompt-out: grid from the 9 units of the other three prompts). The three cat units were
never in the grid and are a true hold-out. Log-ratio vs 2M on the normal grid, ODE:
| units | N | ema m=-0.6 on normal | 2M on grid | ema m=-0.6 on grid | ema_timed m=-0.5 on grid |
| --- | --- | --- | --- | --- | --- |
| 12 old, lpo | 8 | -0.12 [-0.18, -0.06] | -0.52 [-0.70, -0.33] | -0.67 [-0.89, -0.45] | -0.69 [-0.92, -0.47] |
| 12 old, lpo | 12 | -0.08 [-0.14, -0.03] | -0.59 [-0.83, -0.36] | -0.85 [-1.14, -0.56] | -0.83 [-1.12, -0.55] |
| 12 old, lpo | 16 | -0.17 [-0.24, -0.11] | -0.75 [-1.04, -0.48] | -0.93 [-1.27, -0.61] | -0.83 [-1.13, -0.56] |
| 12 old, lpo | 24 | -0.22 [-0.30, -0.13] | -0.76 [-1.13, -0.42] | -0.85 [-1.26, -0.47] | -0.81 [-1.17, -0.47] |
| 3 cat, held out | 8 | -0.10 [-0.16, -0.06] | +0.15 [-0.09, +0.44] | -0.34 [-0.51, -0.20] | -0.35 [-0.52, -0.17] |
| 3 cat, held out | 12 | -0.20 [-0.33, -0.11] | +0.23 [+0.14, +0.39] | -0.24 [-0.31, -0.11] | -0.23 [-0.27, -0.18] |
| 3 cat, held out | 16 | -0.32 [-0.50, -0.01] | +0.06 [-0.04, +0.22] | -0.28 [-0.39, -0.15] | -0.20 [-0.30, -0.04] |
| 3 cat, held out | 24 | -0.28 [-0.53, -0.07] | +0.07 [-0.23, +0.60] | -0.11 [-0.34, +0.28] | +0.03 [-0.15, +0.32] |
(The loo scheme gives the same picture: -0.55 ... -0.93 on the 12 old units.)
- On the four short-tag prompts the grid transfers across prompts: leave-prompt-out error is 40-60% lower
(log -0.5 ... -0.9) and
emaon the grid is better than 2M on the grid by a further 0.1-0.25 in log. - On the held-out
catprompt (a natural-language caption, a different style) the grid does not help 2M (+6% to +26% error at N <= 16) and the combination is no better thanemaalone. With three units this is weak evidence, but it is the only test of the grid outside the prompt family it was derived from, and it fails. - At the settings of the original report (1216x832, CFG 8, Section 7) the grid produced colour artefacts at N = 8. The grid was derived at 576x768, CFG 5, and is not a general schedule.
- The gain of
emaitself (about -10% to -30%) is the part that replicates on every set of units.
7. The settings of the original report (1216x832, CFG 8, prompt cat, 3 seeds)
figures/user_sheet.jpg (first seed = the seed of the failing picture), results/user_check.json; error is
to a 128-step reference of the same kind (ODE) or a 192-step Brownian-coupled reference (SDE), mean over
three seeds. At CFG 8 the trajectories are much more sensitive than at CFG 5 (errors of 0.5-0.9 at 8-16
steps), and three seeds cannot resolve differences below about 15%.
| variant | N = 8 | N = 12 | N = 16 |
| --- | --- | --- | --- |
| ODE 2M, normal | 0.739 | 0.631 | 0.616 |
| ODE ema_timed m=-0.5, normal | 0.693 | 0.656 | 0.603 |
| ODE ema m=-0.6, grid | 0.746 | 0.608 | 0.469 |
| ODE pec gamma=0.5, grid | 1.047 | 0.667 | 0.456 |
| SDE dpmpp_2m_sde, normal | 0.885 | 0.821 | 0.912 |
| SDE ema_timed m=-0.5, normal | 0.946 | 0.972 | 0.885 |
| SDE ema_timed m=-0.5, grid | 1.083 | 0.891 | 0.851 |
| SDE ema m=-0.3, normal | 0.847 | 0.876 | 0.870 |
| SDE extrapolate beta=0.5, normal | 0.816 | 0.781 | 0.856 |
- All ODE samplers on
normalgive clean pictures from N = 12. There is no resolvable difference between them on these three seeds. - The grid breaks N = 8 (colour casts, error 0.75-1.05) and gives lower numbers at N = 16 (0.46-0.47 vs 0.60-0.62) with visible artefacts in some pictures; it should not be used at this setting without re-deriving it.
- With noise (
eta = 1) negative momentum amplifies the injected noise: N = 8 and N = 12 pictures have a blue-violet cast and the error is not lower than fordpmpp_2m_sde. The milderemam = -0.3, which wins in the 15-unit sweep, also gives a blue cast at N = 8 and 12 at this setting (figures/user_sheet_extra.jpg); damped extrapolation (beta= 0.5) does not, and has a lower mean error thandpmpp_2m_sdeat all three budgets (0.82 / 0.78 / 0.86 vs 0.89 / 0.82 / 0.91; three seeds, not significant).
8. Perceptual check (H8)
LPIPS (AlexNet) of the decoded image to the decoded ODE reference, 15 units, ODE, normal grid:
| variant | N | mean LPIPS | 2M | log-ratio vs 2M | wins |
| --- | --- | --- | --- | --- | --- |
| ema m=-0.6 | 8 | 0.451 | 0.515 | -0.131 [-0.257, -0.042] | 14/15 |
| pec gamma=0.5 | 8 | 0.440 | 0.515 | -0.152 [-0.318, -0.027] | 11/15 |
| ema m=-0.6 | 16 | 0.298 | 0.323 | -0.114 [-0.199, -0.037] | 12/15 |
| pec gamma=0.5 | 16 | 0.304 | 0.323 | -0.050 [-0.111, +0.022] | 11/15 |
The latent-L2 winner is also the perceptual winner (H8 supported).
9. SDE sweep (eta = 1, 15 units, equal NFE)
Strong error against the 192-step SDE solution driven by the same Brownian tree (results/sweep_sde.json,
results/sweep.md). Log-ratio against dpmpp_2m_sde (extrapolate_1.0), 95% interval.
| variant | N = 8 | N = 12 | N = 16 | N = 24 |
| --- | --- | --- | --- | --- |
| dpmpp_2m_sde, error | 0.783 | 0.667 | 0.569 | 0.448 |
| core dpmpp_2m_sde (check of the port) | -0.004 | +0.002 | -0.004 | -0.000 |
| extrapolate beta = 0.25 | -0.091 [-0.144, -0.045] | -0.080 [-0.130, -0.035] | -0.113 [-0.164, -0.062] | -0.099 [-0.185, -0.036] |
| extrapolate beta = 0.5 | -0.075 [-0.118, -0.038] | -0.071 [-0.109, -0.034] | -0.107 [-0.164, -0.055] | -0.104 [-0.200, -0.037] |
| extrapolate beta = 0 | -0.089 [-0.142, -0.044] | -0.081 [-0.135, -0.031] | -0.115 [-0.164, -0.065] | -0.070 [-0.129, -0.020] |
| ema m = -0.3 | -0.094 [-0.149, -0.041] | -0.105 [-0.172, -0.047] | -0.157 [-0.212, -0.099] | -0.064 [-0.098, -0.031] |
| ema m = +0.3 | -0.021 [-0.086, +0.034] | -0.015 [-0.094, +0.059] | -0.017 [-0.099, +0.064] | +0.045 [-0.044, +0.132] |
| ema m = +0.6 | +0.093 [-0.005, +0.195] | +0.102 [+0.015, +0.187] | +0.143 [+0.026, +0.254] | +0.258 [+0.115, +0.400] |
| pec gamma = 0.5 | +0.022 [-0.013, +0.062] | +0.010 [-0.014, +0.037] | -0.004 [-0.041, +0.031] | +0.074 [+0.005, +0.156] |
| core dpmpp_sde at N/2 steps (2 NFE per step), error | 0.846 | 0.816 | 0.724 | 0.604 |
| shipped dualsde at N/2 steps, error | 48.1 | 25.5 | 16.3 | 5.8 |
| same, stacked correction removed, error | 1.57 | 1.66 | 1.86 | 1.92 |
Post hoc rows (ph_*), same units:
| variant | N = 8 | N = 12 | N = 16 | N = 24 |
| --- | --- | --- | --- | --- |
| ema m = -0.45 | -0.032 [-0.097, +0.038] | -0.059 [-0.110, -0.002] | -0.173 [-0.234, -0.110] | +0.033 [-0.066, +0.133] |
| ema m = -0.75 | +0.078 [-0.024, +0.184] | +0.024 [-0.085, +0.146] | -0.052 [-0.161, +0.053] | +0.064 [-0.119, +0.216] |
| ema_timed m = -0.5 | +0.015 [-0.069, +0.106] | -0.047 [-0.129, +0.044] | -0.163 [-0.247, -0.083] | +0.007 [-0.120, +0.138] |
| ema m = -0.9 | +0.150 [+0.017, +0.288] | +0.151 [+0.022, +0.289] | +0.113 [-0.011, +0.248] | +0.222 [+0.102, +0.334] |
- In the SDE the gain of strong negative momentum disappears.
m = -0.6is the best ODE variant, but ateta = 1the best coefficient is milder (-0.3) and the best damped extrapolation (beta= 0.25-0.5) is as good. The injected noise is not predictable, and a recursive extrapolation ofDamplifies the part of the error that comes from it. Gains overdpmpp_2m_sde: 6-15% foremam = -0.3 (wins 13/15 at N = 8 and 12, 15/15 at N = 16, 12/15 at N = 24), 7-11% forbeta= 0.25-0.5. - The SDE does not need more damping than the ODE (H5): the best
betais 0, 0, 0, 0.5 for the SDE and 0, 0.25, 0.25, 0.75 for the ODE. - One evaluation per step beats two at equal NFE (H9). The best one-evaluation SDE variant has a
lower strong error than core
dpmpp_sdeat N/2 steps by 16% (N = 8), 27%, 28% and 29% (N = 24); the intervals exclude zero. - The stacked correction of the shipped sampler is not the cause of the failure but it is large (H1):
removing it lowers the SDE error by 60-97% (the effect shrinks with N; at N = 24 one unit improves only
by 9%), but what remains is still 1.9-3.4 times worse than core
dpmpp_sdeat the same NFE because of the variance-exploding noise (Section 4).
10. What to use
All of this is on one model (Anima, bf16) and one resolution/CFG in the main sweeps; Section 7 repeats it at CFG 8, 1216x832 with 3 seeds.
- Deterministic sampling (
eta = 0):ema_timedwithm = -0.5(presetddim_momentum) oremawithm = -0.6. About 11-21% lower error thandpmpp_2mat 8-24 steps, about 1.2x fewer evaluations at equal error; better LPIPS in 14/15 units at 8 steps. Keep|m| < 0.9. - Stochastic sampling (
eta = 1, thedpmpp_2m_sdefamily):extrapolatewithbeta = 0.5(presetdpmpp_2m_sde_damped), 7-10% lower strong error thandpmpp_2m_sdeat 8-24 steps and no colour cast at CFG 8.emawithm = -0.3is slightly better in the sweep (6-15%) but tinted pictures at CFG 8. Strong negative momentum is harmful with noise. - Not recommended: any positive momentum (it only adds lag), the shipped
dpmpp_dualsde_momentumized,res_momentumizedandclyb_4m_sde_momentumizedwere not re-evaluated here but share the same "momentum on increments" definition and the same variance-exploding frame. - The sigma grid (
MomentumSigmasnode) was derived at 576x768, CFG 5 on four short-tag prompts. It transfers across those prompts (Section 6) but not to the natural-languagecatprompt and it broke the 8-step picture at CFG 8; it is shipped as an experiment, not as a default.
11. Limits and what was not done
- One model, one latent size for the sweeps; CFG 5 for the sweeps. The 3-seed check at CFG 8 cannot resolve differences below 15%.
- Reference error is latent L2 to an own high-step solution (5% own error, 24% between two ODE references
for the
faceunit); LPIPS only for the ODE winners and only at 8 and 16 steps. - The strong error of an SDE sampler is not the same thing as image quality; the noise path is shared across methods but a lower strong error does not guarantee a better picture.
- Sections 5 (post hoc rows), 6, 7 and the SDE post hoc rows were not registered. The pre-registered verdicts are in Section 3: 4 supported (H3, H7, H8, H9), 5 refuted (H1, H2, H4, H5, H6).
res_momentumizedandclyb_4m_sde_momentumizedwere not tested.- Bf16 evaluations are not bit-reproducible: the same sampler repeated gave relative errors that differ by
about 0.2% (
core_dpmpp_2m_sdevsextrapolate_1.0).
12. Reproduce
python -m pytest --import-mode=importlib tests (13 tests, CPU)
python run_diagnosis.py (Section 4)
python run_reference.py (references; reuses flow_zero_cost_proxy ones)
python run_sweep.py --eta 0 ; python run_sweep.py --eta 1 (Sections 5, 9)
python run_sweep.py --eta 0 --extended ; python run_sweep.py --eta 1 --extended (post hoc rows)
python run_grids.py --eta 0 ; python export_loo_grids.py ; python run_loo.py --scheme lpo (Section 6)
python run_user_check.py ; python perceptual_check.py (Sections 7, 8)
python analyze.py ; python analyze_grids.py ; python analyze_loo.py lpo
The ComfyUI nodes (SamplerMomentum, SamplerMomentumFilter, MomentumSigmas) and the KSampler entries are in __init__.py;
the dumps/ directory (references, logs) is not versioned.
13. Momentum filter for other samplers (engineering addition, not evaluated)
wrap.py applies the same recursive filter to the data prediction D of any sampler that calls the model
once per step, without touching the sampler: the model is wrapped, Dbar_n = m Dbar_{n-1} + (1 - m) D_n
(ema) or with the time-scaled coefficient m (sigma / sigma_max - m / 2) (ema_timed); the last step
receives the unfiltered D. Registered in the KSampler list as <base>_momentumized:
- deterministic bases (
ema_timed, m = -0.5):euler,lms,dpmpp_2m,ipndm,ipndm_v,deis,res_multistep,gradient_estimation; - stochastic bases (
ema, m = -0.2):euler_ancestral,ddpm,dpmpp_2m_sde,dpmpp_2m_sde_gpu,dpmpp_2m_sde_heun,dpmpp_2m_sde_heun_gpu,dpmpp_3m_sde,dpmpp_3m_sde_gpu,res_multistep_ancestral,er_sde,sa_solver.
The node Sampler Momentum Filter (any sampler) exposes the base sampler, mode, strength, eta and
s_noise. Multi-evaluation samplers (heun, dpm_2, dpmpp_sde, dpmpp_2s_ancestral, seeds_*,
uni_pc) are excluded.
None of this was evaluated. The defaults are chosen by analogy with Sections 5 and 9. The only observation on the real model is a one-off smoke run at 8 steps (576x768, CFG 5, one seed): lms_momentumized and ipndm_v_momentumized returned a latent with std about 10 at the default strength (their unfiltered versions: about 3), which is a sign that -0.5 is too strong for them; at 16 steps the values were 1.95 and 2.12 against 1.32 and 1.28. A
negative filter on top of a multistep sampler that already extrapolates may over-extrapolate; tune
strength with the filter node and treat positive strengths as harmful. Native presets were renamed:
dpmpp_2m_momentum is now ddim_momentum and dpmpp_2m_sde_momentum is now dpmpp_2m_sde_damped.
14. Prior work and what is new
Checked on 2026-11-14 by searching arXiv, GitHub and the ComfyUI-Manager node list. I read abstracts and README files only; I did not read the code of these packages, did not run them and did not compare against them. The search is not exhaustive.
Papers:
- Wizadwongsa, Chinchuthakun, Khungurn, Raj, Suwajanakorn, Diffusion Sampling with Momentum for Mitigating
Divergence Artifacts (arXiv:2307.11118, ICLR 2024; code
sWizad/momentum-diffusion). Heavy Ball momentum inside existing numerical solvers (PLMS and others) to enlarge their stability regions, plus a generalised variant (GHVB). This is the closest work. Their aim is fewer divergence artifacts at low step counts (pixel and latent diffusion models, FID-type metrics); ours is the error against a converged solution on a flow-matching model, and the sign of the coefficient. I did not check which sign conventions they scan, so the finding here that a positive coefficient only adds lag is not a refutation of theirs: the model family, the solver and the metric differ. - Wang, Dinh, Liu, Xu, Boosting Diffusion Models with an Adaptive Momentum Sampler (arXiv:2308.11941): an Adam-style momentum sampler with adaptive update.
- Jang, Youn, Jeon, Lee, Rethinking Peculiar Images by Diffusion Models: Revealing Local Minima's Role (AAAI 2024): naive and positive-negative momentum in diffusion sampling, aimed at peculiar images.
- Chen, Zheng, Berthelot, Gu, Susskind, Zhai, TADA: Improved Diffusion Sampling with Training-free Augmented Dynamics (arXiv:2506.21757): momentum dynamics through higher-dimensional initial noise, a different mechanism.
ComfyUI-Manager entries that mention momentum:
ComfyUI Extra Samplers(Clybius): the*_momentumizedsamplers studied above.ComfyUI-sonar(blepping): momentum-based "Sonar" sampling for Euler, Euler Ancestral and DPM++ SDE.Seren Sampler(seren-alegro): momentum-predicted, tolerance-controlled sampling for rectified-flow models of the Anima family, one model evaluation per accepted step; its README names two earlier packs, "FlowMomentum" and "AdaptiveFlow", which I did not find.DDRK Omega Sampler(HVOSTOVSKY): a universal flow-matching and EDM sampler with a momentum option.
What I could not find, and therefore think is new here (a claim about my search, not about the field):
- An explanation of why the shipped
dpmpp_dualsde_momentumizedfails on a flow model (variance-exploding formulas, a stacked correction, an overwritten parameter), with an executable port checked against the original (Section 4). - A measurement, at equal evaluation count and against converged ODE and SDE references, of how the sign
and size of an exponential-moving-average momentum on the data prediction change the error, including
its stability edge (
|m| < 1) and the sign flip between a positive coefficient (lag) and a negative one (extrapolation) (Sections 5, 9). - The observation that the same filter wraps any one-evaluation sampler (Section 13, untested).
What is not new: momentum and Heavy Ball ideas in diffusion sampling (above), and the exponential-integrator frame, which is DPM-Solver++ as implemented in ComfyUI core. The size of the effect is modest (11-21% lower error, about 1.2x fewer steps, one model).