Apply OmDet

Apply OmDet is the one-node version of "find the things I name in this image." Drop in an image, type the things you care about as a comma-separated list, and it hands you back three things at once: the image with red boxes drawn on it, a mask per detected object, and a Labelme-style JSON of every box. No training, no fixed class list - if you can write the word, it'll try to find it.

That last part is the point. ComfyUI's ecosystem is stuffed with segmentation tools (SAM, BiRefNet, and friends), but most of them answer "what's the subject here?" OmDet answers "where is this specific thing I named?" It's open-vocabulary detection: the model hasn't memorized a fixed list of 80 COCO classes, it matches whatever words you type against the image using a text encoder. If your workflow ever needs to selectively edit a named object - swap the jacket on the person, redraw the cat, cut out the car for compositing - this is the step that turns a word into a mask.

How it actually works

Under the hood it's a wrapper around OmDet from om-ai-lab, specifically the OmDet-Turbo_tiny_SWIN_T model: a DETR-style detector with a Swin-Tiny vision backbone plus a CLIP ViT-B-16 language backbone. The node turns your comma-separated prompt into a sentence - "Detect person,cat,dog." - and runs inference at 640×640 internally. Then it draws the boxes and labels (with confidence, so you see person:0.87 on the image), fills each box into a binary rectangle mask, and packs everything into a Labelme dict per image.

Two things worth knowing before you wire it in. First, the masks are rectangles around the detections, not fine segmentation shapes - plot_boxes_to_image fills the box with cv2.rectangle. If you need pixel-perfect edges, run the MASK through SAM or a BiRefNet pass afterward. Second, the model is re-loaded on every execution - the code builds a fresh inference engine per call, so the first run of a workflow (and every re-run) takes a beat to load weights.

The inputs that matter

Of the five inputs, a beginner sets three:

• prompt - comma-separated labels, default person,cat,dog. The comma is the only separator that matters; the node even normalizes Chinese commas for you.
• conf_threshold (default 0.3) - how confident the detector must be to keep a box. Lower it and you get more hits and more noise; for a single clear subject you can push it to 0.5.
• device - cuda or cpu. CPU works, but it's slow.

The outputs are IMAGE (annotated copy - wire that to a preview or Save Image), MASK (one rectangle mask per detection, concatenated - route it into Set Mask for inpainting or compositing), and JSON (a list of Labelme dicts, one per input image, with label, points, and shape_type: rectangle per box - handy if you're building a dataset or an annotation pipeline).

Installing it

Install via ComfyUI Manager (search "ComfyUI-OmDet") or clone it:

cd ComfyUI/custom_nodes
git clone https://github.com/prodogape/ComfyUI-OmDet

Then restart ComfyUI. Models download themselves on first run - two weights (OmDet-Turbo_tiny_SWIN_T.pth and the ViT-B-16.pt CLIP backbone) pulled from Hugging Face into ComfyUI/models/OmDet/. Give the first run a minute.

Here's the gotcha the README hides: the shipped requirements.txt is conda-flavored. Running pip install -r requirements.txt will fail - pytorch-cuda doesn't exist on PyPI, and detectron2 isn't on standard PyPI either. Don't fight it. Your ComfyUI environment already has torch and torchvision, so install only what's actually missing:

pip install lmdb
pip install detectron2 -f https://dl.fbaipublicfiles.com/detectron2/wheels/cu118/torch2.0/index.html

Match the wheel index to your torch/cuda build. On Windows this is the fiddly part - detectron2 wheels are picky about torch versions, and the common failure mode is a missing or mismatched wheel, not your code. If the node throws an import error mentioning detectron2, that's it.

Troubleshooting

Beyond the detectron2 wheel dance: a hang on first run is usually the model download, not a crash - watch for the Downloading... line in the console. If you're getting boxes all over the place, your confidence threshold is too low; raise it before you touch NMS. And remember the rectangle-mask caveat - if your inpaint seams are blocky, that's the box mask showing through, and you want a segmentation pass on top rather than more NMS tuning.

It's a niche node with a real job. For "find the named thing and give me a mask," it's one node where GroundingDINO-style setups take several. Just budget a little patience for that first install.