Eval Sampler (Advanced)

The name flatters it. EvalSampler - labeled "Eval Sampler (Advanced)" in the node menu - is not a better sampler and won't improve a single image you make. It's a wiretap. At every denoising step it writes the model's conditioned and unconditioned predictions to disk as safetensors files, then a companion script turns those recordings into a numbers table. You run it on a baseline model and again on every variant you're curious about - fp8, GGUF Q4 or Q8, an INT8 build, a LoRA you're stress-testing - and find out exactly how far each one drifts from the baseline instead of squinting at two renders and arguing.

This is the measurement tool behind the same author's "INT8 in the age of MXFP8" quantization shootout (that's Robert Johnson, the INT8-Fast / Flux2-INT8 person). It exists because eyeballing samples doesn't cut it once the differences get small. So the honest use case is narrow but real: you're evaluating quantized checkpoints, or writing it into a test harness, and you want numbers.

How it works

EvalSampler is a subclass of ComfyUI's built-in KSamplerAdvanced - same inputs, same output, same sampling behavior. The one addition: it installs a hook on the model's CFG function. Every step, the hook grabs cond_denoised and uncond_denoised (the two predictions CFG amplifies the difference between), saves each as a safetensors tensor, then hands off to the real CFG function and lets sampling continue as if nothing happened.

Files land in ComfyUI/output/eval_latents (the output_folder input) named {seed}_S{step}.safetensors for the conditioned prediction and {seed}_S{step}_uncond.safetensors for the unconditioned one. Keep the seed the same across runs and the files line up one-to-one, which is exactly what the comparison needs.

The inputs that matter

Everything else is a normal KSamplerAdvanced: model, add_noise, noise_seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, start_at_step, end_at_step, return_with_leftover_noise. Only two inputs are actually yours to think about:

• noise_seed - the recording's index. Identical seed across every run you want to compare, or the pairing breaks.
• output_folder - name of the subfolder under ComfyUI/output. Give each variant its own folder; the analysis script keys on folder names.

The LATENT output is real and usable - the node still produces the image, so it's a literal drop-in for KSamplerAdvanced in a working graph.

Reading the results

After collecting runs, analyze from the pack directory:

python MeasureResults.py --eval-dir <ComfyUI>/output/eval_latents --stratify-std

It expects a baseline subfolder named BF16, then compares every other subfolder against it. You get a color-coded table of MSE, rel-RMSE, SNR in dB, cosine similarity, per-channel error, variance ratio and outlier fraction, with the best value per row starred - plus analysis_results.csv, a GitHub-flavored comparison_table.md, and an MSE-vs-file-size plot if matplotlib is installed. --stratify-std replaces pooled standard deviations with per-timestep standard errors, which are far more meaningful for a mean estimate, so keep it on. (Rough translation of the headline numbers: SNR dB is "how loud is the signal vs. the noise the quantization added," cosine sim is "are the latents pointing the same way," rel-RMSE is "how wrong, as a percentage of how big the values are.")

Installing

ComfyUI Manager, search "EvalSampler" or the pack title ComfyUI-EvalSampler. Or the manual route:

cd ComfyUI/custom_nodes
git clone https://github.com/BobJohnson24/ComfyUI-EvalSampler

then restart ComfyUI. There's no requirements.txt - it only uses things ComfyUI already ships (torch, safetensors, folder_paths); matplotlib is needed only for the plot. One real dependency hidden in there: the module imports comfy_api.latest at the top, so you need a reasonably recent ComfyUI or neither node in this pack will show up at all.

Where people get burned

The comparison is only as fair as the runs feeding it. Same prompt, same steps, same sampler, same scheduler, same cfg, same seed - change any of it between runs and you're measuring your own sloppiness, not the model.

The nastier trap is CFG 1. ComfyUI skips the unconditioned pass entirely at exactly 1 (it's a free speed win unless a sampler opts out), and the plain EvalSampler doesn't opt out - so at CFG 1 your _uncond files are filled with zeros and the whole uncond half of the analysis is garbage. Run the eval at a real CFG like 5–8 on SD 1.5/SDXL. That also means this tool is aimed at classic CFG-driven models, not guidance-distilled ones that live at 1 by design.

Finally, it writes two files per step per batch element. Thirty steps, batch of 4, and you've just dumped a few hundred small safetensors - harmless, but it adds up fast, so clean the folder between test rounds.