Zero-Shot Image Classification

This is the node that turns ComfyUI into a self-checking machine. ZeroShotImageClassificationPipeline takes an image, takes a list of labels you invent on the spot, and returns a JSON ranking of how well each label fits. "cat, dog, bird" by default, but the labels are yours: "portrait, landscape, abstract," "photorealistic, illustration," "your character, someone else." It's CLIP doing what CLIP does best - comparing an image against text descriptions - and it's arguably the most useful node in kadirnar/ComfyUI-Transformers, because it plugs the "is this the image I wanted?" feedback loop directly into your graph.

The pack wraps Hugging Face's transformers.pipeline() as one node per task. Default model is openai/clip-vit-base-patch32 - the classic CLIP, small (~350 MB) and fast enough to run on CPU without suffering.

How it works

Mechanically: pipeline("zero-shot-image-classification", model=model_name) embeds the image and each label text into a shared space, then returns similarity-ranked results as JSON. Your candidate_labels string gets split on commas and trimmed, so "cat, dog, bird" becomes three labels. The output results_json is a formatted array like [{"label": "cat", "score": 0.81}, ...], best first.

Two things worth internalizing:

• Scores are relative to your label list. CLIP normalizes across the labels you give it, so a 0.7 for "cat" only means something next to the other labels you typed. Add "mammal" to the list and the cat score drops.
• Only the first frame is considered, same as the pack's other image nodes. A batch in, one frame used.

Inputs and outputs

• image - an IMAGE tensor, straight off a VAE decode.
• candidate_labels - STRING, comma-separated. The whole trick is here: descriptive phrases ("a red sports car") often beat terse words ("car") at discriminating.
• model_name - default openai/clip-vit-base-patch32; free text. clip-vit-large-patch14 is bigger and better if you have the VRAM.
• Output: results_json - STRING.

Wire the image in from any sampler's VAE decode. To read the JSON you need a Show Text node; to branch on the result, parse it for the top label.

Installing it

Pack-standard install - ComfyUI Manager (search "ComfyUI-Transformers") or:

cd ComfyUI/custom_nodes
git clone https://github.com/kadirnar/ComfyUI-Transformers
cd ComfyUI-Transformers
pip install -r requirements.txt

then restart. The README's cd custom/nodes is a typo; requirements drag in transformers>=4.42 and the usual stack. First run downloads CLIP from the Hub to ~/.cache/huggingface.

Common issues

• Weak scores across the board. Usually a label-wording problem - CLIP matches descriptions, so "a photo of a cat" beats "cat" for realistic images. Rewrite labels, not settings.
• Slow first run, then slow-ish every run. Model download first, then a per-execution model load (this pack's nodes skip its own cache). Not real-time, but fine for a per-generation check.
• It judges style, not quality. CLIP knows "photorealistic vs. illustration" reasonably well but can't tell you an image is good. Don't overreach on what the scores mean.

The honest take: if you only keep one node from this pack, make it this one. A generate → classify → decide loop (re-roll if the top label doesn't match what you asked for) is the single most practical trick in the entire pack, and it costs you nothing but one small model download and a few seconds per image.