VQ Workflow Comparison

The pack's documentation advertises this node as doing t-tests and Wilcoxon signed-rank tests - the kind of rigorous "is workflow B actually better than A, or was that lucky seed?" statistics you'd want before making decisions. Here's the honest part: the code doesn't run any statistical tests. The README oversells it. What the node actually does is parse two JSON metrics blobs and compare them key-by-key, printing the delta and a "winner." That's a diff, not a significance test. Useful, but know what you're getting.

Still, the use case is real: you ran the same prompts through two workflows, you have metric JSON for each, and you want a quick side-by-side report instead of squinting at two text outputs. For a rapid "which one looks better on paper" pass, this is fine. For decisions that matter, do the stats yourself - the pack even ships the utilities for it, they're just not wired into this node.

How it works

It takes two JSON strings, parses them, and iterates over every key. Numeric values get compared: A's value vs B's value, the delta, and a winner flag. Non-numeric values just get printed side by side. The output is a plain-text report you can pipe into a display or save node. Simple, deterministic, zero heavy lifting.

Inputs and outputs

Required:

• metrics_a_json (STRING) - workflow A's metrics as JSON.
• metrics_b_json (STRING) - workflow B's metrics as JSON.

Output:

• comparison_report (STRING) - the formatted A-vs-B report.

Installing

Part of ComfyUI-VideoQuality-Metrics:

cd ComfyUI/custom_nodes
git clone https://github.com/jajos12/ComfyUI-VideoQuality-Metrics
pip install -r ComfyUI-VideoQuality-Metrics/requirements.txt

Restart ComfyUI; it's under VideoQuality/Reporting. Manager installs the pack too.

Common issues

• "Error: Invalid JSON input" - one of the strings isn't valid JSON. The logger's output is valid JSON, but see the next point.
• It expects flat key→number JSON, but the logger emits nested JSON. The logger organizes metrics under fidelity/temporal/distributional objects; this node compares top-level keys, so dict values just get printed as strings and nothing useful happens. If you want these two nodes to work together, feed it a flattened object ({"psnr": 32.1, "fvd": 120.5, ...}) instead of the logger's output.
• No p-values, no confidence intervals. The doc's claims about "P < 0.05 means it's real" don't apply to what this node actually does. For a proper test, collect the per-run scores and run a t-test or Wilcoxon yourself - the numbers will mean something then.