H3 Studio · Smart Benchmark Lab

Every serious H3 user eventually has the same conversation: "Does LightX v1.0 at 8 steps actually beat Base Balanced at 12? Is 4-step PDD worth the extra LoRA wiring?" The honest way to answer is a controlled run - same prompt, same seed, different recipes - and the normal way to do that is a pile of A/B workflows you'll never look at again. H3StudioSmartBenchmark (H3 Studio · Smart Benchmark Lab) replaces that pile with one node that builds the whole comparison matrix from a single Director context.

What it actually is

A guarded benchmark matrix rather than a hard-coded A/B graph. It takes your h3_bundle (from the H3 Studio Loader) and studio_context (from the Director), plus a scenarios_json you build in the frontend with real pickers - each named scenario can vary the transformer model, Sampling Profile, runtime preset, resolution and an ordered stack of custom LoRAs. Because everything hangs off one Director, the prompt, references and seed stay fixed across cells, which is the entire point: you're isolating the thing you changed.

How it works

The node expands your scenarios into a plan and - this is the good design - tells you the exact run count before anything queues. Oversized matrices get blocked unless you explicitly flip allow_large_matrix, which is a genuine save on a 33B video model where every cell is a full generation. During execution it reports the active cell, completed/remaining counts, profile, aligned resolution, elapsed time and an ETA once enough cells exist to estimate one honestly. Optional live_cell_previews shows each result as it lands; disable them for maximum throughput. At the end you get a comparison_grid image and a text benchmark_report with per-cell timings.

The inputs that matter

There are a lot of them, but the ones you'll actually touch:

• seed_strategy - three choices that change what the matrix means. Same seed for all is the fair apples-to-apples comparison; New seed each row gives paired results; New seed every image is a diversity sweep where each cell gets its own seed.
• repeats and max_generations - repeats per cell (up to 16) and the total cap (up to 128). The node reports the real count before you commit, so treat max_generations as your budget.
• allow_large_matrix - the safety valve. Leave it off until the node complains.
• compare_vae - appends the pack's identical-latent original-vs-image-VAE isolation test after your main plan, handy if you're deciding whether the experimental T=1 decoder is worth it.
• include_reference_context / include_original_prompt - whether the final comparison sheet carries the references (with correct @ImageN labels) and the source prompt. Defaults are on, and you want them on for a meaningful sheet.

Outputs: comparison_grid (an IMAGE you can preview or save) and benchmark_report (a STRING of the run's verdicts and timings).

Install

It's part of the H3 Studio pack - ComfyUI Manager (search ComfyUI-MiniMax-H3-Studio) or:

cd /path/to/ComfyUI/custom_nodes
git clone https://github.com/thaakeno/ComfyUI-MiniMax-H3-Studio.git
cd ComfyUI-MiniMax-H3-Studio
python -m pip install -r requirements.txt

Restart and hard-refresh. To benchmark PDD profiles you also need the separately installed ComfyUI-MiniMaxH3-PDD-Mamad8 package, plus the PDD LoRA and heads in models/loras/ and models/pdd_heads/. LightX profiles need the matching LightX H3 LoRAs in models/loras/.

Where people get burned

The matrix math is the trap. "12 scenarios × 3 resolutions × 2 repeats" is 72 full H3 generations, and on a 33B model that's not an afternoon. Read the run-count report before you queue, keep max_generations sane, and don't flip allow_large_matrix just because it's there. Also remember PDD is REF2VA-only - benchmark profiles that need a reference context with no references wired up, and the pack rejects the combination rather than silently mis-running. This pack is alpha and classic-frontend-only; Nodes 2.0 support hasn't landed.