VLM Judge (one image, one question)

Want to know whether a generated image actually matches what you asked for - no API key, no cloud, no per-call cost? VLMJudge runs a vision-language model on your own GPU and asks it one question about one image. It's the QA counterpart to the face-similarity node this pack is named after: that one checks "same person?", this one checks anything you can put into words - "Rate how well this follows the prompt, 1–10, as JSON."

The design is where it earns the "judge" name. General captioning nodes append a system prompt to whatever you type, which is fine for captions and wrong for judging - a template glued on the end quietly competes with your actual question. VLMJudge sends your prompt verbatim, nothing prepended or appended, because a judgement has to be reproducible from the prompt alone. And it's deterministic by default: temperature 0 means greedy decoding, so the same image and prompt give the same answer every run.

What it does under the hood

It loads the model straight from HuggingFace using whatever repo id you type. The load path is where the fiddly bits hide: FP8 models carry their own quantization config, so it loads with dtype="auto" (which respects that config), uses sdpa attention (the only backend FP8 weights work with here), and loads straight onto the GPU. It applies the model's chat template, then returns only the new tokens the model generated, not the echoed prompt.

The model stays resident between calls (keyed by model id + device) so a batch of questions doesn't reload each time - which is why the pack also ships VLMUnload: ComfyUI's /free can't evict a model a custom node is holding.

Inputs and outputs that matter

• image - one image. Batches aren't the point.
• prompt - your whole instrument. Put the rubric here. Empty prompt returns a "no prompt given" message.
• model_id - default Qwen/Qwen3-VL-8B-Instruct-FP8. Any transformers-compatible vision repo works; swap in a smaller Qwen3-VL variant for less VRAM.
• temperature (default 0) - greedy decoding, as the tooltip says: same image and prompt, same answer.
• seed - only matters once temperature is above 0.
• max_tokens (default 384) - how long an answer the model is allowed.
• keep_loaded (default true) - hold the model resident for the next call, or drop it at the end of this run.
• require_json - when on, the node parses the answer as JSON (stripping code fences) and reports whether it parsed.Outputs: response (the answer string) and is_json (whether it parsed, when you asked for JSON). Both also land in /history, since it's an output node, so API callers read the verdict without a second node. The optional device input flips between cuda and cpu, with an automatic fallback to CPU if CUDA isn't actually available.

Install and the deps nobody documented

Search ComfyUI_Face_similarity in ComfyUI Manager, or:

cd ComfyUI/custom_nodes
git clone https://github.com/phobod/ComfyUI_Face_similarity.git
cd ComfyUI_Face_similarity
pip install -r requirements.txt

Here's the trap, and it's a real one: the README only documents the InsightFace face-similarity node, and the pack's requirements.txt only installs that stack - insightface, onnxruntime-gpu, opencv. VLMJudge needs transformers (imported at runtime, not at load), and the FP8 default additionally needs the kernels package. Neither is in requirements.txt. If your ComfyUI env already has transformers - most do, via other packs - the first run just downloads the model (several GB, needs internet) and goes. If not, you'll hit a No module named 'transformers' error at the first judge call, not at install. pip install transformers fixes it.

The pack also quietly handles one nasty FP8 failure for you. ComfyUI sets HF_HUB_DISABLE_TELEMETRY=1 at startup; with telemetry off, the kernels library builds a malformed user-agent that makes HuggingFace's httpx reject the request. The visible symptom is "could not verify publisher trust status", which reads like a permissions problem and isn't one. This pack patches the header at import, so that confusing failure shouldn't bite you.

Gotchas worth knowing

• Budget VRAM honestly. An 8B model is not free. The console logs how much it holds (resident: 9.1 of 12.0 GB). On a small card, set keep_loaded = false per run or point model_id at a smaller Qwen3-VL variant - or use cpu, and go make tea.
• An LLM/VLM node is arbitrary Python with full OS access - the category that once shipped malware in this ecosystem. This pack is small, MIT and readable, so glance at it before trusting a fresh repo-id model download.
• The prompt is the instrument. Ask for JSON and get prose? That's require_json returning false - the model ignored your format instruction, which is itself data about whether it's usable for your QA loop.