VLM Detections to Masks

A bounding box is a rectangle someone drew around the truth. A mask is the truth. When your detection node returns boxes or polygons and your next step needs pixel-level control - inpaint just the object, composite it onto a new background, or inspect exactly what the detector saw - VLM Detections to Masks is the converter that makes it happen.

It's a workhorse utility in gokayfem's VLM_nodes pack. Detection feeds in as VLM_DETECTIONS; seven outputs come out covering every mask flavor a workflow might want. Given how much of ComfyUI's ecosystem is mask-driven (inpainting, IP-Adapter, compositing, ControlNet-ish region control), this node is where detected objects become editable.

How it works

For each detection it rasterizes the mask - using the polygon or quad when the detection has one, otherwise falling back to the bounding box. Masks come out as both a per-frame union (all objects together) and individual per-object masks, plus an inverse of the union. The union and individual masks are also converted to viewable black-and-white IMAGE previews, and there's a mask_map_json mapping every mask back to its detection record - label, score, frame, and which output index it landed on.

The last output, instance_maps, is the underrated one: a stable-color map where each detected object gets its own consistent color. It's the fast way to see that the detector separated the person from the car, instead of squinting at overlapping boxes.

Inputs and outputs

• detections (VLM_DETECTIONS) - the only input.

Outputs, in order: union_masks, individual_masks, mask_map_json, inverse_union_masks, union_mask_images, individual_mask_images, instance_maps. For most workflows you'll grab union_masks (to inpaint everything detected) or individual_masks (to isolate one object), and instance_maps for the visual check.

Installing this pack

It's one pack, one install. ComfyUI Manager: search VLM_nodes. Or:

cd ComfyUI/custom_nodes
git clone https://github.com/gokayfem/ComfyUI_VLM_nodes
cd ComfyUI_VLM_nodes
python -m pip install -r requirements.txt

Use ComfyUI's Python; this node itself downloads nothing (though whatever detector produced your detections may have).

Common issues

The honest limitation: masks from detection boxes are rectangles, not contours. If a detection came in as a bare box, the mask is a rectangle - feed it to VLMMaskProcessor to feather/grow/shrink it, or better, get polygon or SAM2.1 masks upstream so the rasterization has real edges to work with. Also watch batch shapes: union_masks is one mask per frame, individual_masks is per-object-per-frame, and the pack's README reminds you mask tensors stay in-process - they don't appear in any JSON export, by design.