[Inference.Core] SAM Segmentor

Meta's Segment Anything Model (SAM) is best known as an interactive tool - click a point or draw a box on any image and it segments the object under it with impressive precision. This node uses SAM differently: no clicking, no bounding box, no text prompt. It automatically segments an entire image into flat regions and hands you the result as a ControlNet preprocessor output - a segmentation-style map that gives the diffusion model a layout of what goes where (sky here, building there, person here) without specifying what any of it should actually look like.

That's a genuinely different job from ControlNet's more common conditioning types. Depth and pose control spatial arrangement and body position; segmentation controls category layout. It's a niche but real condition type, and this pack's SAM preprocessor is one entry in that category alongside OneFormer and UniFormer, which are the more commonly used segmentation preprocessors elsewhere in the ecosystem.

Worth knowing before you build around this

Segmentation ControlNet is one of the condition types that didn't survive the shift to newer base models. It was part of the 2024-era xinsir SDXL union - genuinely one of twelve conditions bundled into a single capable checkpoint - but the union models trained since for Flux, Qwen-Image, and Z-Image dropped it, along with normal maps, Reference conditioning, and QR/brightness patterns. If you're planning a segmentation-conditioned workflow on a current-generation base and can't find a checkpoint for it, that's not a setup problem - SDXL's union model is genuinely still the only mainstream place this condition type lives.

Don't confuse this with SAM used as a masking tool ahead of inpainting, which is a much more common thing to see SAM doing in ComfyUI graphs - Impact Pack ships its own SAM-based detector providers for exactly that job (finding and masking a specific region for FaceDetailer-style targeted fixes), and that's a different use case from this node's role as a ControlNet segmentation-map generator. If you're trying to isolate one object for inpainting rather than condition a whole generation on a layout, Impact Pack's SAM tooling, or SAM paired with GroundingDINO for text-prompted object selection, is the more common route.

Inputs and outputs

There's not much to tune. image is required, and the only optional input is resolution (64–2048, default 512), the standard preprocessor resize control every node in this pack shares. Output is a single IMAGE - the segmentation map - which wires into a Segmentation-type ControlNet alongside your usual checkpoint and prompt.

Installing it

Ships with the LykosAI Inference Core Nodes pack. Via ComfyUI Manager, search "ComfyUI-Inference-Core-Nodes". Manually:

cd ComfyUI/custom_nodes
git clone https://github.com/LykosAI/ComfyUI-Inference-Core-Nodes

then install.py, or pip install -e .[cuda] (swap in [cuda12], [rocm], or [directml] for your hardware) and restart, rather than the plain pip install -e ., for GPU-accelerated inference.

Troubleshooting

With only one real input, there's not much to adjust if a segmentation map looks off. In practice, the more common fix is switching preprocessors entirely - if you have comfyui_controlnet_aux installed as well, OneFormer or UniFormer are worth comparing against this node's output before assuming your source image is the problem.