SAM3 Character Agent (Local)

"Agent" in this node's name isn't marketing. This is a genuine loop: a vision-language model looks at your image, guesses simple phrases SAM 3 can segment, actually runs them, looks at the masks with its own eyes, and keeps iterating until it's satisfied. No API key, no cloud - a GGUF vision LLM running next to the segmenter, all local.

SAM 3's open-vocabulary segmentation is genuinely good, but shallow. It nails "blonde woman" or "red dress"; it has no idea what "the fourth woman from the left holding a suitcase" is. That gap - between what SAM 3 knows and what you actually need masked - is exactly what this node exists to bridge. Hand it an image and a description, and it composes that description out of the simple noun phrases SAM 3 does understand.

The honest take first

The author, Adam Barbato, is refreshingly blunt about his own work: he also built the ComfyUI-Sa2VA node, and in his release post he says the agentic approach is "often worse (and much slower)" than purpose-trained Grounded SAM or Sa2VA, and that he personally sticks with Sa2VA for its speed. Where this earns its keep is the 100%-agentic, no-API, fully-local case - and the moment you need a mask for something no trained detector or SAM 3 phrase covers. Local 8–30B VLMs also hallucinate enough that the "check the masks" step sometimes cancels out its own benefit. Set expectations accordingly.

How it works

Each iteration, the LLM picks one or two-word noun phrases (the system prompt pushes hard toward "no articles, no possessives, no numbers"), SAM 3 segments them, and the node renders every accumulated mask as a numbered, color-coded overlay on the original image. That image goes back to the LLM, which either calls accept_masks with the mask numbers it wants or calls segment_phrases again with better guesses. Repeat until accepted or max_iterations runs out. The "brain" is loaded via llama-cpp-python with a vision chat handler, which is why you need both a GGUF model and its mmproj vision projector.

Inputs that matter

• image - the frame to segment.
• character_description - be specific; the default "A person with brown hair" is just a placeholder. The richer your description, the better the agent's phrase decomposition.
• llm_model / mmproj_model - dropdowns populated from ComfyUI/models/llm_gguf/. Pick a vision-capable GGUF and its matching mmproj; Gemma 3 27B and Qwen 3 VL 30B are the ones the author found workable.
• max_iterations (1–20, default 5) - how many guessing rounds. More chances to converge, more time.
• confidence_threshold (0.1–1.0, default 0.5) - how confident SAM 3 must be about a mask.

Outputs

• mask (MASK) - the combined binary mask for your character. Feeds everything a mask feeds: SetLatentNoiseMask for inpaint/regional conditioning, ApplyMaskToImage for compositing, or a Detailer loop if you convert it to SEGS.
• debug_images (IMAGE) - one frame per segmentation attempt. Preview these the first run; they're the fastest way to see which phrases the model wasted on.

Installing it

Via ComfyUI Manager (search "ComfyUI-Segmentation-Agent") or:

cd ComfyUI/custom_nodes
git clone https://github.com/adambarbato/ComfyUI-Segmentation-Agent
cd ComfyUI-Segmentation-Agent
pip install -r requirements.txt

Restart ComfyUI. Two things beyond the clone: the requirements.txt pulls heavy deps (llama-cpp-python, timm, pycocotools, triton), and the model files. Drop sam3.pt (from huggingface.co/facebook/sam3) into ComfyUI/models/sam3/, and your GGUF + mmproj into ComfyUI/models/llm_gguf/ - the same folder Searge's LLM pack uses.

Where people get burned

• The dropdowns show "none". That's the node telling you models/llm_gguf/ is empty - it raises "Please ensure both LLM model and mmproj files are present" if you run it anyway. The mmproj file is the one everyone forgets.
• Vision support. Stock llama-cpp-python wheels lag on newer vision models. The README points at JamePeng's fork with prebuilt wheels; on Windows the extra blocker is triton, which the SAM 3 stack needs and which is notoriously fiddly there.
• VRAM. The node keeps SAM 3 and the GGUF LLM resident, so a 27–30B model alongside the segmenter wants a serious card. It does cache models between runs and unloads on demand.
• It's slow, inherently. The first iterations frequently burn guesses on phrases SAM 3 can't parse at all. If it's not converging, watch the debug output before just cranking max_iterations - sometimes the fix is a better description, not more rounds.