modaux: sam

Meet the heavyweight of this pack. modaux: sam runs Meta's Segment Anything Model (SAM) on your image and hands back a segmentation map where every detected object - person, tree, car, cat - is painted a flat, random color. That map is the "segmentation" ControlNet condition: the model reads the colored blobs as layout only and fills each one with whatever your prompt says it should be. The colors carry zero meaning; where each blob sits and what shape it is, that's the entire signal.

It's the most visually striking preprocessor output in the whole pack, and the most expensive one to run. If you've seen the "draw a composition in MS Paint, get a photoreal render" workflow, this is the node doing the heavy lifting.

How it works

The node wraps SamDetector from controlnet_aux. On first run it downloads sam_vit_h_4b8939.pth - the giant ViT-H checkpoint, roughly 2.5 GB - from ybelkada/segment-anything on Hugging Face. SAM runs automatic mask generation over your image, the results get sorted by area, and each mask region is filled with a random RGB color. No pretrained ControlNet is involved; SAM is pure image segmentation.

Two things worth knowing before you run it:

• It's slow. ViT-H is a big model, and automatic mask generation over a full image is one of the heavier preprocessor jobs in the ecosystem. Start with detect_resolution at 512 and expect to wait.
• The colors are random on every run. The same image twice gives you a different color palette both times, which is fine - again, the layout is the condition, not the hues.

The node has no optional inputs. Just image, detect_resolution, image_resolution (both default 512), and one IMAGE output. Wire that into a segmentation ControlNet: SDXL's xinsir union supports "segment," and Z-Image has a community-trained SAM ControlNet. On SD 1.5 the control_v11p_sd15_seg checkpoint is the usual target. There is no segmentation ControlNet for Flux-family bases as of mid-2026, so like the normal-map nodes in this pack, this is an SD 1.5 / SDXL play.

Why you'd reach for it

Segmentation conditioning is the fastest way to a deliberate composition. Instead of fighting a prompt to place three things in a scene, you paint three blobs in the right spots and the model respects that layout while the prompt decides the contents. It's also the node people use when they want to reuse a scene's geometry on a totally different subject - same blob layout, new prompt, entirely new image.

Installing it

ComfyUI Manager: search ControlNetAux (repo madtunebk/ComfyUI-ControlnetAux). Or by hand:

cd ComfyUI/custom_nodes
git clone https://github.com/madtunebk/ComfyUI-ControlnetAux
cd ComfyUI-ControlnetAux
pip install -r requirements.txt   # timm, controlnet-aux==0.0.7, mediapipe

Restart ComfyUI; the node lives under the ControlNet Auxiliar category.

Where people get burned

Budget the first run: the 2.5 GB SAM download plus ViT-H inference on a 512px image is the single heaviest first-run experience in this pack - on a modest GPU, give it minutes, not seconds. And keep the identity straight: the small beta "ControlNet Auxiliar" pack by madtunebk is not Fannovel16's comfyui_controlnet_aux (the pack behind AIO_Prep in shared workflows). If you're only here for SAM, know that SAM also powers the masking side of tools like the Impact Pack's FaceDetailer - this node specifically produces the colored segmentation map for ControlNet, not a mask. They're different jobs; this one is for composition.