NNT Visualize Confidence Scores

Accuracy tells you your model is right 95% of the time. It doesn't tell you whether it's sure when it's right, or wildly guessing when it's wrong. NNT Visualize Confidence Scores is the node that answers that second question: feed it a tensor of confidence scores - the probability the model attached to its chosen class for each sample - and it renders a plot of their distribution, with your decision threshold drawn right on the picture.

It's the inference-side counterpart to NntVisualizeTrainingMetrics. Training plots tell you if the model learned; this node tells you how it feels about its answers. In the pack's workflow, the confidence scores usually come from the inference/evaluation side of a trained model (the softmax probabilities it assigns to each prediction).

How it works

The main input is confidence_scores, a TENSOR of per-sample scores. Then you pick a plot_type:

• histogram - the distribution of scores, with your threshold as a red dashed vertical line.
• scatter - one point per sample on a score axis, threshold as a horizontal line.
• box - a box-and-whisker summary of the scores, threshold line again.
• combined - all three stacked in one figure. The default, and the one you want for a first look: you see the shape of the distribution, the per-sample spread, and the summary stats in one glance.

threshold (default 0.5) is where you draw the line between "confident" and "hesitant." It doesn't change the model - it's a reference line for your eye, so you can see at a glance how many predictions fall on the wrong side of a decision boundary. image_width and image_height (default 1024×768) size the output image.

Outputs: an IMAGE (the rendered plot, ready for Preview Image / Save Image) and a STRING (a text summary of the stats).

Where people get burned

Two things. First, the node calls .numpy() on the tensor directly, so it expects a CPU tensor - if your confidence scores are still on CUDA, it can throw. Keep the scores on CPU (the inference nodes in this pack generally produce CPU tensors, but check if you're hand-building one). Second, this expects confidence scores - probabilities in roughly [0, 1] - not raw logits. Feeding it raw model outputs will produce a plot that looks meaningless because your "confidence" values live outside the range you set the threshold in. Softmax first.

The useful reading: if your histogram is bimodal - a hump near 1.0 and another near 0.5 - your model is confident when right and suspiciously unconfident when wrong. That's the pattern worth acting on: gather more data around the uncertain region, or reconsider the threshold.

Install

Pack-level install:

cd ComfyUI/custom_nodes
git clone https://github.com/inventorado/ComfyUI_NNT.git
cd ComfyUI_NNT
pip install -r requirements.txt

Restart ComfyUI (or Manager → "ComfyUI Neural Network Toolkit NNT"). Needs matplotlib and seaborn, both in the pack requirements (torch, numpy, pandas, sklearn, transformers, statsmodels, onnx, shap 0.41.0, …). Heavy install, slow first run, but once it's in, this is a genuinely instructive five-minute addition to any inference graph.