VLM Open-Vocabulary Detection

Closed-vocabulary detectors know the classes they were trained on and nothing else. VLM Open-Vocabulary Detection is the opposite: you type the thing you're looking for - "red car," "mug with a handle," "person in a blue jacket" - and it finds it. No retraining, no class list, just a label prompt and boxes out.

It's the entry point to the detection side of gokayfem's VLM_nodes pack, and it deliberately hides four different detectors behind one stable interface so your workflow doesn't care which one you picked. The four models:

• Grounding DINO Tiny (fast) - the default-ish starting point; small and quick.
• Grounding DINO Base - the higher-quality sibling.
• OWLv2 Base Ensemble - strong zero-shot for lists of concepts.
• OmDet Turbo Swin Tiny (fast) - real-time-oriented and efficient.

All coordinates come out in source-image pixels, in the pack's versioned VLM_DETECTIONS socket, so every downstream utility (filter, render, crop, masks, points) just works.

How it works

You give it an image (a still or a full video frame batch - it processes batches frame by frame) and a labels string of concepts separated by commas, semicolons, or newlines. It runs detection, applies box_threshold and text_threshold to cut low-confidence junk, and caps results at max_detections. The fps input matters only for video: connect the fps output of GetVideoComponents so every timestamp in the result is correct.

It outputs a lot on purpose: detections (the typed socket), json (serialized), preview (IMAGE with boxes drawn), box_mask (MASK), and both a single bounding_boxes (core BOUNDING_BOX) and bounding_boxes_with_metadata (BOUNDING_BOXES). One node feeds renderers, mask pipelines, SAM2 segmentation, and core ComfyUI region logic.

The optional inputs worth knowing: nms_threshold for suppressing overlapping duplicates, precision (auto/bfloat16/float16/float32), batch_size (frames per model call - keep 1 for low VRAM, raise it when memory allows), and unload_after (true frees the model after the run if another big model needs the card next).

Installing this pack

One pack, one install. ComfyUI Manager: search VLM_nodes. Or:

cd ComfyUI/custom_nodes
git clone https://github.com/gokayfem/ComfyUI_VLM_nodes
cd ComfyUI_VLM_nodes
python -m pip install -r requirements.txt

Run pip with ComfyUI's Python. Unlike the pure-utility nodes, this one does download weights: the first execution pulls the selected detector (Grounding DINO Tiny is the lightest; OWLv2 and OmDet are heftier) into ComfyUI's model directory, so give the first run a few minutes and an internet connection.

Common issues

Threshold tuning is where everyone spends their first hour. text_threshold (default 0.25) is the one that usually needs raising - a "person 0.19" false positive pollutes everything downstream; bump it toward 0.3–0.4 and pair it with VLMFilterDetections for a second pass. For video, forgetting to wire fps from GetVideoComponents gives you wrong timestamps on every detection - the README calls this out explicitly. And VRAM: leave batch_size=1 on a 8GB card; the pack's own memory strategy says start with Grounding DINO Tiny and only scale up the model once the pipeline is correct. For tracking-by-detection, run this over the complete bounded batch and feed detections into VLMTrackDetections.