Yogurt InstructSAM Image Segmentation

This is the node that does what the old "GroundingDINO finds it, SAM cuts it out" two-step promised but in a single pass. You feed it an image, type something like Please segment the two blue ceramic mugs in the image., and it comes back with a mask per mug, a merged mask, and the count. No point clicks, no boxes, no separate detector model to download. That's the whole pitch, and it's real - InstructSAM is a vision-language model that fuses Qwen3-VL (the understanding half) with SAM3 (the masking half), so a sentence becomes instance masks end to end. It's the one you reach for when you need to mask a specific one-off object - the yellow fruit behind the apples, that jacket in the corner - and no community YOLO detector exists for it. That's exactly the gap promptable segmentation was built to fill, per the modidex masking docs: faces and hands get a trained detector, everything else gets a sentence.

How it actually works

The model treats your instruction as a chat message. The Qwen3-VL processor wraps the image plus your text into a prompt, the model generates text output and a stack of mask logits with a confidence score per candidate instance, and the node post-processes from there. The two thresholds are the entire control surface, and they do different jobs:

• score_threshold (default 0.3) - drops candidate instances whose score isn't strictly above this. Missed a mug? Lower it. Getting junk objects you didn't ask for? Raise it.
• mask_threshold (default 0.5) - the binarization cutoff on the mask probabilities. Lower it and masks grow wider; raise it and they hug tighter.

Each surviving instance becomes one binary MASK in instance_masks, and union_mask is all of them OR'd together - the usual thing you actually wire downstream into a blur, an inpainting pass, or a SaveImage preview. You also get text (the model's textual answer), scores (a JSON string of the kept instances' scores), and instance_count (an INT, handy for branching logic). If nothing clears score_threshold, you get an all-black mask at image size, an empty scores array, and a zero count - no crash, just nothing found. Oh, and it's strictly one image at a time and RGB only; hand it a batch tensor and it refuses.

Setting it up

It's half of a two-node pack, so install is the shared story: ComfyUI Manager (search "Yogurt InstructSAM") or

cd ComfyUI/custom_nodes
git clone https://github.com/yogurt7771/ComfyUI-YogurtNodes-InstructSAM
cd ComfyUI-YogurtNodes-InstructSAM
python -m pip install -r requirements.txt   # Windows Portable: ..\..\python_embeded\python.exe -m pip install -r requirements.txt

then restart ComfyUI fully (browser refresh doesn't reload Python nodes). The only real dependency is transformers>=4.57.3,<6. Wire the loader's model output into this node, load an image, type your instruction, run. A Load Image → this node → SaveImage preview of union_mask is the fastest first run.

Where people get burned

Instruction quality is 90% of the results, and thresholds can't save a bad one - they only filter candidates and adjust edges, not fix a misunderstood sentence. The README is explicit: use specific English full sentences with category, color, position, or relation ("the yellow fruit behind the apples", not "fruit"). Short, vague, or Chinese prompts are measurably less stable; the model was trained mostly on English. If you're missing objects, lower score_threshold before you rewrite anything - but if the mask is in the wrong place entirely, rewrite the prompt. And if you switched model files mid-session, the loader may hand you a stale cached handle; re-run with force_reload=true on the loader rather than fighting a phantom.

One honest caveat: this is a 2B model doing a big job, so the masks are good but not surgical-grade against a fine-tuned SAM 3 setup, and there's barely any community chatter about InstructSAM yet - you're early, so treat the thresholds as tuning knobs you'll actually have to turn.