SAM3 Character Agent (OpenRouter)

This is the same agent loop as the pack's Local node, with one swap that changes everything: the vision-language "brain" lives on OpenRouter instead of on your GPU. SAM 3 still runs locally - the segmentation is always on your machine - but the part that looks at the image, proposes the simple phrases to segment, and then judges the results is now a cloud model. And per the author's own testing, this is the variant that can actually win.

He's candid about why in the README: the agentic approach only beats one-shot Grounded SAM or Sa2VA when the vision model is good enough to rarely hallucinate, because the whole trick is trusting the LLM to check the masks it gets back. Frontier Gemini models clear that bar; smaller local ones often don't. So if your segmentation target is image-specific ("the fourth woman from the left holding a suitcase") and you want the best possible accuracy, the OpenRouter version is the one he recommends - you're paying for the model's eyes, not for the cutting.

How it works

Identical machinery to the Local node: each iteration the LLM calls segment_phrases with a few one-to-two-word noun phrases, SAM 3 masks them, the node renders all accumulated masks as a numbered overlay, feeds that image back, and the LLM either calls accept_masks with the right numbers or tries again - up to max_iterations. The only difference under the hood is the request: the node POSTs to https://openrouter.ai/api/v1/chat/completions with your key, 2048 max tokens, temperature 0.7, and a 300-second timeout. No llama-cpp-python, no GGUF, no VRAM for the LLM.

Inputs that matter

• image - the frame to segment.
• character_description - be vivid; the default "A person with brown hair" is a placeholder.
• openrouter_api_key - your key from openrouter.ai (they give free credits to start).
• model_name - the OpenRouter model ID. Must be a vision model: it has to read the image and the mask overlays. The default in the node is google/gemini-2.0-flash-exp:free; the README and the bundled example workflow both point at google/gemini-3-flash-preview for best quality.
• max_iterations (1–20, default 5) and confidence_threshold (0.1–1.0, default 0.5) work exactly as in the Local node.

Outputs

• mask (MASK) - the combined mask, ready for SetLatentNoiseMask, ApplyMaskToImage, or a Detailer loop.
• debug_images (IMAGE) - one frame per attempt, so you can watch the agent think.

The pack's example workflow shows the natural shape: LoadImage → this node, with the mask feeding both MaskPreview and ETN_ApplyMaskToImage, and the debug batch going to a PreviewImage.

Installing it

Same pack, same clone - via Manager (search "ComfyUI-Segmentation-Agent") or:

cd ComfyUI/custom_nodes
git clone https://github.com/adambarbato/ComfyUI-Segmentation-Agent
cd ComfyUI-Segmentation-Agent
pip install -r requirements.txt

The good news: you only need one model file, sam3.pt (from huggingface.co/facebook/sam3) in ComfyUI/models/sam3/. No GGUF, no mmproj. One wrinkle: the shared requirements.txt still pulls in llama-cpp-python even though this node never imports it - harmless if you never use the Local node, but it's the one dependency with the prebuilt-wheel hassle.

Where people get burned

• The placeholder key. Run it with the default sk-or-v1-... and the node raises "Please provide a valid OpenRouter API key" immediately.
• Picking a text-only model. model_name is a free-form string, so it's easy to paste a non-vision model and get confusing failures - the image messages go nowhere. Stick to the recommended Gemini vision models.
• Every iteration is an API round-trip. At default 5 iterations you can easily burn several calls per image, and the free tier rate-limits. It's still cheaper than the VRAM a 30B local model would cost you, but it's not free forever.
• The 300-second timeout. A slow overloaded model can hit it mid-run. Cancelling mid-loop returns whatever masks accumulated so far, which is a nice touch.