Advanced Anime Segmentation (ONNX)

SimpleAnimeSeg is fast but dumb about edges: when your character's hair or outfit melts into the background you get a mask full of holes and background crumbs. AdvancedAnimeSeg is the same pack's answer - a two-stage pipeline that detects the character first, then runs a dedicated mask refiner over the rough cutout. The "advanced" in the name means flexible and multi-stage, not guaranteed-better; the README itself warns that the simple model can win in specific edge cases. But when you need a genuinely clean anime character mask, this is the node in this pack you'd actually reach for.

How it works. Stage one is an RTMDet-based instance segmenter, CondInst-style: the model emits a mask prototype plus per-instance coefficients, and the node assembles the actual mask with a few small dynamic convolutions - all pure PyTorch, so there's no mmcv/mmdet dependency dragging in. It scans three feature strides (8/16/32), picks the single highest-confidence detection, maps that box back to your original coordinates, and binarizes the instance mask with threshold. That "single detection" part matters: one character per call. Group shots? It segments whoever it's most confident about and ignores the rest.

Stage two, on by default, is the refiner: enable_refine feeds your image plus that coarse mask - concatenated into a 4-channel input - into an ISNet-based model that cleans up the edges, binarized by refine_threshold. It uses the converted weights from Faor-Mati/anime-character-segmentation, which in turn come from dreMaz's AnimeInstanceSegmentation.

The inputs that matter:

• threshold (default 0.3) - how aggressive the detector's instance mask is. Raise it to shed background junk, lower it to keep thin parts.
• refine_threshold (default 0.3) - the refiner's binarization. This is the one to nudge for hair.
• enable_refine (default on) - turn it off and the node skips stage two; both outputs become identical.

The outputs: Refined Mask and Coarse Mask, both MASK. Wiring the refined one into a ConditioningSetMask or ControlNet inpaint is the standard move; invert either for background replacement. Having both is genuinely useful - you can see exactly what the refiner changed and decide which edge you trust.

Installing it

Same pack as SimpleAnimeSeg, but the advanced node wants all three ONNX files, ~590 MB total:

cd ComfyUI/custom_nodes/comfyui_animeseg
mkdir -p models
curl -L -o models/anime_segmentor_rtmdet_e60_simplified.onnx \
  https://huggingface.co/Faor-Mati/anime-character-segmentation/resolve/main/anime_segmentor_rtmdet_e60_simplified.onnx
curl -L -o models/mask_refiner_isnetdis_refine_last_simplified.onnx \
  https://huggingface.co/Faor-Mati/anime-character-segmentation/resolve/main/mask_refiner_isnetdis_refine_last_simplified.onnx

The isnetis.onnx file from the simple node isn't used by this one, but you'll want it anyway since the pack has both nodes. ComfyUI Manager installs the code fine; it won't fetch these weights. Only new Python dependency is onnxruntime.

Troubleshooting

• It's slow, and it's CPU-only. Both models run through CPUExecutionProvider, hardcoded in the source - onnxruntime-gpu won't help. Two models per image, so budget several seconds per frame.
• Hair is still ragged: lower refine_threshold, not threshold - the refiner is what touches fine edges.
• It picked the wrong character in a group shot: not a bug, it's the single-best-detection design. Crop to the character you want first.
• You'll see [Refine] Logits range: ... spam in the console. Harmless debug prints left in the code; ignore them.
• Quality is worse than the simple node: possible, and the README says so. When the model misses, try SimpleAnimeSeg on the same image and pick by eye.

And the same warning that applies to the whole pack: this is a per-image editing tool. Don't point it at a LoRA training dataset - a background-stripped dataset teaches the model to generate blank backgrounds.