H3 Studio - Benchmark Lab

Which H3 sampling profile is actually better - Base Quality at 20 steps, LightX at 8, the 4-step ER-SDE recipe? Most people answer by running two generations and squinting at the outputs, which is a terrible way to compare because the seeds differ, the resolutions drift, and you can't remember which one you liked. H3StudioABComparison is the pack's answer: a guarded comparison matrix that runs profiles × resolutions under controlled conditions and hands you a side-by-side grid.

The name undersells it - it's not just A/B, it's a full matrix. comparison_kind offers two shapes:

• Sampling profiles x resolution (default) - compare any two (or more) sampling profiles across a set of resolutions.
• VAE decode - same T=1 latent - sample one latent once, then decode that exact latent through the original H3 video VAE and optionally the T=1 image VAE. This isolates decoder behavior instead of mixing sampling variance into the result. Genuinely clever, and the right tool if you're trying to decide whether the T=1 image VAE is worth its softness.

The inputs that keep you out of trouble

• profiles and megapixels - comma/newline-separated lists. "Two profiles remain a simple A/B," per the description; add more for a wider matrix.
• repeats and seed_strategy - the fairness control. "Same seed for all - fair comparison" is the fairest A/B; "New seed each row - paired comparison" keeps Base/LoRA paired; "New seed every image - diversity sweep" is explicitly not a fair test, just a diversity tour. seed_step offsets between row/image seeds.
• max_generations (default 24) and allow_large_matrix - the guard. The node reports the exact run count before queueing and rejects oversized matrices unless you explicitly check the box. This is the "benchmark without destroying your weekend" feature: it's easy to type 3 profiles × 4 resolutions × 3 repeats and accidentally ask for a 36-generation run.
• include_reference_context / include_original_prompt - whether the final sheet shows the references (with correct @ImageN labels) and the source prompt.
• live_cell_previews (on) - shows completed cells in the node; disable for maximum throughput.

Outputs and how it reports

comparison_grid (IMAGE) is the assembled contact sheet; comparison_report (STRING) carries the per-cell metadata - profile, seed, requested vs real dimensions, repeat, sampling time, and the rest. The node reports the active cell, completed and remaining counts, profile, aligned resolution, elapsed time, and an ETA once enough cells exist to estimate one honestly. It also dedupes native-capped variants when profile, seed, dimensions and prompt are identical, so you don't pay twice for the same work.

For the report, the Lazy output switch is the intended partner: it makes sure a benchmark run never also fires the normal sampler branch (and vice versa).

Install

cd ComfyUI/custom_nodes
git clone https://github.com/thaakeno/ComfyUI-MiniMax-H3-Studio.git
cd ComfyUI-MiniMax-H3-Studio
python -m pip install -r requirements.txt

Restart and hard-refresh the frontend. No extra dependencies. Start with two profiles, one resolution, same seed - that's the honest A/B, and it's what the defaults are shaped to encourage.