Xiaomi-Lens-0 Segment

Every mask you didn't paint by hand came from something that decided where. YOLO decides by class, SAM from a point or box you hand it, Grounded SAM by gluing a text detector to SAM - and since late 2025, SAM 3 does text natively and has been the default path.

Xiaomi-Lens-0 Segment is the same idea with a 4B vision-language model underneath, and that changes what you can ask. Not just "the pink dress" - though it does that, one mask and one box per instance - but Which object in the image would you use to stir the drink?, and it comes back with the spoon. It reasons, then commits to a mask. Different tool from a detector, and it's why this is worth 9 GB.

How it gets from a sentence to a mask

It's a Qwen3.5-4B with a pixel-mask decoder bolted on. Your image goes in resized to a multiple of 32 pixels (the processor caps at 8.4 MP), the text follows it in the chat template, and generation runs greedy - do_sample=False, as upstream's infer.py. Then the pack replays prompt plus completion and decodes each <|mask|> token into a probability map on the encoder's canvas: sigmoid, threshold, resize back to your image.

Two gotchas follow. Masks inside a <think> block are discarded - deliberation isn't the answer. And results repeat exactly within one setup but not across setups; Transformer versions move the low digits, with masks agreeing at IoU 0.975+ between 5.4.0 and 5.20.1.

The inputs

model comes from the Loader, image is your IMAGE, and query is a multiline string defaulting to the cat.

prompt_mode is the interesting one. On find, your text is dropped into the grounding prompt - Locate all instances in the image that match the description "<query>". Report bbox coordinates and masks. - so a short phrase is all you write. On instruction, it goes through exactly as typed, which is what questions need; the shipped reasoning_segment.json asks which part to replace to change the season to winter and gets the grass.

threshold defaults to 0.5, upstream's value; 0 gives you the raw probabilities instead of a binary mask, which is what you want for a soft blend. max_new_tokens caps generation at 16384 - leave it room, because cost follows the answer's length. The 23-coin example took 29 seconds on a 5090 mostly because the answer was long, not because 0.1 MP is hard.

The outputs, and how each one wires up

• mask - every instance merged into one. Send it to Grow Mask and an inpainting sampler, or to Invert Mask and Join Image with Alpha for a cut-out (the shipped cut_out.json).
• instance_masks - a mask batch, one per instance, in the order the model answered them. Per-object edits run on this, and preview's colours follow the same order.
• preview - your image with palette-coloured overlays and box outlines. Look at it before you trust a mask.
• text - the whole answer, special tokens and all, exactly as infer.py prints it. Useful on a reasoning run, mostly scaffolding on a find run.
• boxes - a JSON string, not a box type: each entry has the image index, the box in pixels of your image, and the same box in the model's 0–1000 scale.

That last one is the trap: boxes won't plug into anything expecting a bounding box, so print or save the string if you want them. To act on a region, use mask or instance_masks.

Install

The pack has two nodes, so this is the whole install. ComfyUI v0.38.0+, Python 3.10+, Transformers 5.4.0+.

cd ComfyUI
git clone https://github.com/hiroki-abe-58/ComfyUI-XiaomiLens custom_nodes/ComfyUI-XiaomiLens
python -m pip install -r custom_nodes/ComfyUI-XiaomiLens/requirements.txt
python custom_nodes/ComfyUI-XiaomiLens/tools/download_model.py --models-dir models/xiaomi_lens

Then restart and open workflows/gui/segment_object.json - it runs on ComfyUI's own example.png, so you can check the install without hunting for a test image. The pack is unofficial and unaffiliated with Xiaomi (code MIT, vendored model code and weights Apache-2.0).

Where people get burned

There is no error when the model finds nothing. You get zeroed masks and the queue carries on, which reads like success - so a run that "worked" and produced nothing to inpaint is the normal shape of a bad query, not a bug. Check preview on the first runs of a new query. The same silence covers a truncated answer: if max_new_tokens cuts the completion off before the model emits its mask token, you get empty masks rather than a complaint, so leave the default headroom and, on instruction, ask for masks explicitly the way the shipped examples do.

VRAM is the real ceiling: 9.5 GB at 1 MP, 11 GB at 3 MP, 15 GB at 8 MP peak. If your sampler wants the card too, set the Loader's after_run to move to CPU so the segmenter hands its ~9 GB back. And keep the honest limitation in mind - this is a judgment call, not a lookup, so it will sometimes miss an object or merge two into one mask. For faces and hands at scale a YOLO detector is still faster and better.