Zero-Shot Object Detection

Zero-Shot Object Detection finds objects you describe in words, without ever training on them. Type "cat, dog, person," feed it an image, and out come bounding boxes with labels and confidence scores - drawn directly onto a copy of the image. The "zero-shot" part is the point: classic detectors are locked to a fixed class list, but this node lets you name whatever you're looking for at runtime.

The default model is google/owlvit-base-patch32, Google's open-vocabulary detector built on the OWL-ViT architecture. It pairs a CLIP-style text encoder with a vision backbone, so your candidate labels get embedded as text and matched against image regions. The node wraps the zero-shot-object-detection pipeline from Hugging Face: it splits your comma-separated labels, runs detection, draws red boxes with label (score) text on the image, and returns the detections as JSON.

The inputs that matter

• image - the IMAGE tensor to scan.
• candidate_labels - comma-separated words to look for, default cat, dog, person. These are the whole game; be specific ("red car" beats "car") but don't over-engineer.
• threshold - minimum confidence to draw a box, default 0.1, range 0–1. This default is low - expect noise. Bump it to 0.3–0.5 for cleaner output.
• model_name - Hub model ID as a string, default google/owlvit-base-patch32.

Two outputs: annotated_image (an IMAGE with boxes and labels drawn on) and detections_json (a STRING with label, score, and box coordinates for every detection).

How to install it

Standard pack install - ComfyUI Manager (search "ComfyUI-Transformers") or:

cd ComfyUI/custom_nodes
git clone https://github.com/kadirnar/ComfyUI-Transformers

Restart ComfyUI. OWL-ViT base is about 600MB on first download, and the pack's requirements.txt covers the transformers stack.

Where people get burned

The threshold. At the shipped default of 0.1, OWL-ViT boxes everything that even vaguely matches, and beginners rightly assume the node is broken. Start at 0.3 and tune from there. The other gotcha is label phrasing - OWL-ViT matches text to visual concepts, so labels should read naturally ("person on a bike" is fine, "bike-person" is not). And keep expectations realistic: it's a solid open-vocabulary detector, but it's not as sharp as a fine-tuned specialist model on its home domain. For finding "a thing I can name but have no model for," though, this is exactly the right tool.