OneFormer COCO Segmentor

This node paints your image by category. Every pixel gets labeled - person, car, sky, road, chair - and colored accordingly, producing a semantic segmentation map for a seg ControlNet. The point of segmentation conditioning is that it gives the model a layout of what goes where without saying anything about how those things look. You're handing over the composition as labeled blobs and letting the prompt decide the appearance.

How it works, and COCO vs ADE20K

OneFormer is a strong segmentation model, and this is the variant trained on the COCO dataset - roughly 133 common object and "stuff" categories. It runs a single pass that assigns each pixel a class and maps that class to a fixed color. A seg ControlNet (control_v11p_sd15_seg) is trained on exactly this color-coded convention, so it reads the map and generates a person where the "person" color is, sky where the "sky" color is, and so on.

The pack ships two OneFormer segmentors and the difference is the training set. COCO (this node) is tuned for common objects and people - reach for it when your scene is object- or person-centric. The ADE20K variant covers ~150 scene-parsing classes and leans more toward architecture and environments. If your image is a room, a building, or a landscape, ADE20K often labels it more sensibly; for a person-and-things scene, COCO. Try both if you're unsure; they cost the same to run.

The inputs and output

Refreshingly simple:

• image - your source.
• resolution (default 512) - processing size; match it to your render.

The single output is an IMAGE - the color-coded segmentation map - which wires into a ControlNet Apply node with a seg ControlNet. Inert until that model is loaded.

Installing it

ComfyUI Manager: search ComfyUI's ControlNet Auxiliary Preprocessors, install, restart. Or:

cd ComfyUI/custom_nodes
git clone https://github.com/Fannovel16/comfyui_controlnet_aux
pip install -r requirements.txt

Restart ComfyUI. First run downloads the OneFormer checkpoint from HuggingFace - OneFormer is a chunky model, so expect a real download and a heavier compute footprint than a simple edge detector.

Where people get burned

First, the colors are a language, not decoration. The seg ControlNet expects the specific color-to-class mapping OneFormer uses, so don't hand-edit the segmentation map's colors arbitrarily and expect the model to follow - if you recolor a "building" blob to a random hue, the ControlNet no longer knows it's a building. If you want to edit the layout, edit which regions are which class, keeping the canonical colors.

Second, model weight and speed. OneFormer is one of the heavier preprocessors here. If you just need loose spatial guidance and segmentation feels like overkill, depth or a rough scribble might get you there faster.

Third, the landscape note: semantic segmentation is one of the ControlNet conditions that largely stayed in the SD1.5 and SDXL era and never got rebuilt for the newest base models. So a seg-ControlNet workflow is comfortably an SD1.5/SDXL move. That's fine - just don't go looking for a matching seg model on the latest architectures, because it isn't there. And as always, the map does nothing without the seg ControlNet loaded and a ControlNet Apply node wired in.