DiffusionGemma Tally Audit

Here's a phenomenon you only see with a diffusion LLM: the model can commit to a wrong answer early, while the canvas is still hot, and then spend the rest of the run polishing it - the annealing equivalent of being too confident to change your mind. DGemmaTallyAudit is a narrow, specific tool for catching exactly that. It audits a "count the numerals" task run - the benchmark-style prompt where you ask the model how many digits are in some text - against the model's own restated evidence, frame by frame.

Why that task? Because it has a checkable ground truth that appears inside the model's own output. As the canvas denoises, each frame shows the model's running tally claim. This node parses every frame's claim, tracks how the tally changed between frames, and then checks the final claim for arithmetic consistency against the numbers the model itself restated. The output is an audit report telling you whether the model's per-step claims agree with its final answer - and if a numeral's count was revised mid-run, exactly when.

How it works

One input: frames - the sampler's decoded STRING list (its frames output). Not the trace, not the canvas state - the already-decoded per-step text. The node needs the whole ordered list at once so it can compare frame over frame, which is why it uses ComfyUI's list-input convention rather than running once per frame.

Output: audit_report (STRING). It gives you, per frame, a parse status (ok, partial with the unparsed numerals named, or unrecognized); a list of revision events (numeral 3: 4 -> 5, step 12 -> 15); and a final verdict: whether the final tally is arithmetically consistent with the model's own restated evidence. That last line is the one worth reading - INCONSISTENT means the model's claimed count disagrees with the numbers it itself wrote down, which is the smoking gun for the "committed early, stayed wrong" failure mode.

The honest framing

This is a niche node attached to a specific probe, not a general-purpose truth checker. It doesn't validate that the answer is correct - it checks internal consistency between the model's per-step claims and its final restatement. If the model confidently wrote down the wrong count and restated it consistently, the audit will say "consistent" and you've learned something subtler: the anneal was stable, not correct. That distinction is exactly the kind of thing this pack was built to surface.

Install

Same pack, same story: install ComfyUI-DiffusionGemma once (Manager search the title, or clone into custom_nodes, restart, transformers==5.13.0 + diffusers>=0.39.0), wire a sampler's frames output in, and read the report. You'll use it rarely - but the day you suspect a diffusion LLM of confidently wrong arithmetic, it's the difference between a hunch and evidence.