LayerMask: MaskEdgeUltraDetail

This is one of the nodes people install the whole pack for. You feed it your image and a rough mask, and it refines the edge using actual matting - the thing that recovers hair, fur, and fuzzy boundaries instead of leaving you a hard, contaminated cutout. If you've ever gotten a decent-but-not-great mask out of a segmentation model and wished the edges were cleaner, this is the fix-up pass.

The distinction that matters, and the one the background-removal literature hammers on: a segmentation mask labels each pixel foreground or background - a hard yes/no. That's structurally wrong for a wisp of hair or a soft edge, where the truth is "60% subject." Matting predicts that fractional alpha. This node runs your rough mask and image through a matting solver to turn a crunchy edge into a soft, correct one.

How it works

It uses the image as a guide to re-solve the alpha along the mask boundary. Two backends: PyMatting (a proper closed-form matting solver - higher quality, slower, needs the pymatting package) and OpenCV-GuidedFilter (a faster guided-filter approximation). Internally it grows/adjusts the mask into a trimap-like transition band, then refines within it. The output is both a cleaned image and a cleaned mask.

The inputs and outputs that matter

• image (IMAGE) and mask (MASK) - the source and the rough matte. The image guides the refinement, so they must line up.
• method - PyMatting (quality) or OpenCV-GuidedFilter (speed). Reach for PyMatting when the edge is the point; guided filter when you're batching and need it fast.
• detail_range (INT, 1–256, default 12) - how wide the refinement band is around the edge. Wider catches more stray hair but risks pulling in background.
• black_point / white_point (FLOAT, defaults 0.01 / 0.99) - clamp the alpha ends so semi-transparent junk resolves to fully-out or fully-in.
• mask_grow (INT), fix_gap (INT, 0–32), fix_threshold (FLOAT, default 0.75) - pre-shape the mask: grow/shrink it, close small holes, and set how aggressively gaps get filled before matting runs.

Two outputs: image (IMAGE, the refined cutout) and mask (MASK, the refined alpha). Usually you want the mask to drive a composite.

How to install it

ComfyUI Manager: search ComfyUI Layer Style, install, restart. Or:

cd ComfyUI/custom_nodes
git clone https://github.com/chflame163/ComfyUI_LayerStyle

Restart so the pack's requirements.txt installs - the PyMatting backend depends on the pymatting package coming in that way.

Common issues & troubleshooting

If PyMatting mode errors out but the node otherwise loads, the pymatting dependency didn't install - reinstall the pack (or pip install pymatting into ComfyUI's Python), or just switch method to OpenCV-GuidedFilter, which has lighter requirements.

The pack-level version of this is IMPORT FAILED at startup, meaning ComfyUI_LayerStyle didn't load at all. Across this pack that's usually a stale install from before it split into ComfyUI_LayerStyle and ComfyUI_LayerStyle_Advanced (delete and reinstall), or a broken transformers/tensorflow dependency crashing the import chain. A clean reinstall via Manager clears it.

On results: if it drags background into the edge, narrow detail_range and push white_point down / black_point up to firm up the alpha. PyMatting is genuinely slow on big images - that's expected, not a hang; use guided filter for previews and PyMatting for the final. And this refines an existing mask - it won't invent one. Garbage mask in, garbage-with-nicer-edges out.