LayerMask: MaskEdgeUltraDetail V3

This is the node you add when your cutout looks fine until you zoom into the hair. Background removers - BiRefNet, rembg, SAM, whatever - are great at finding the subject and terrible at the last mile: flyaway strands, fur, motion blur, semi-transparent edges. As the community keeps rediscovering, the quality frontier moved years ago from "can it find the subject" to "can it survive hair," and nobody's fully won that. MaskEdgeUltraDetail V3 is chflame163's answer: hand it your rough mask and it re-does the edge with proper image matting, turning a blocky selection into a soft, strand-accurate one.

The way it works is trimap matting. It takes your coarse mask, erodes it inward to mark "definitely subject," dilates it outward to mark "definitely background," and treats the band between as unknown - then it solves the actual alpha values in that band. That's why the results survive hair: instead of a hard yes/no per pixel, you get real partial transparency where a strand half-covers a pixel.

The inputs and outputs:

• image and mask - your source image and the rough mask to refine.
• method - the matting engine, and the biggest decision here. VITMatte is a transformer matting model and gives the best hair quality (it downloads a model on first use; VITMatte(local) and vitmatte-base-composition-1k are variants for a locally-placed copy). PyMatting and GuidedFilter are classical CPU algorithms - lighter, no model download, faster, lower ceiling. Start with VITMatte on GPU; fall back to GuidedFilter if you can't download the model.
• The trimap knobs - mask_grow, and the erode/dilate/blur controls (transparent_trimap_erode, transparent_trimap_dilate, trimap_blur) plus black_point / white_point - shape how wide the "unknown" band is and how the matte is contrasted. Wider band = more room to recover fine detail, but slower and more prone to leaking background.
• device (cuda/cpu) and max_megapixels (default 3) cap where and how big it runs.

Two outputs: image (your source) and mask - the refined, hair-accurate matte, which is the thing you actually want. Wire that mask into your composite or background replacement.

Where it fits: it's a refinement stage, always downstream of an actual segmentation. The pattern is: BiRefNet/rembg/SAM makes the rough mask → MaskEdgeUltraDetail V3 cleans the edges → you composite. It's the difference between a cutout that looks pasted and one that looks shot on the new background. For product shots or hard-edged subjects it's overkill, but for people, animals, and anything with hair or fur, it's the node that saves the shot.

Practical notes: VITMatte needs its model, so first run may pause to download, or fail if the box is offline - that's when the PyMatting/GuidedFilter fallbacks earn their place. It's compute-heavy, hence the megapixel cap; run it on GPU. And it can only refine an edge that's roughly right - if the input mask misses whole chunks of the subject, fix the segmentation first, because this node polishes edges, it doesn't find subjects.

Install is the pack standard. Easiest is ComfyUI Manager - search ComfyUI Layer Style (or "LayerStyle"), install, restart. Manual: cd ComfyUI/custom_nodes && git clone https://github.com/chflame163/ComfyUI_LayerStyle, then pip install -r requirements.txt from inside the folder using ComfyUI's Python, and restart. On a hosted ComfyUI it's usually preinstalled, VITMatte model included.

The pack-wide caveat, and it bites hardest on nodes like this one: Layer Style is notorious for import failures - install it, restart, and the whole node set shows up red or missing. It's a dependency clash. The pack pulls a heavy stack (transformers, onnxruntime, opencv), and one mismatched version - a stray transformers or tensorflow being the classic - takes the entire pack down on import. Since VITMatte rides on transformers, this node is especially sensitive to that. Fix it by reading the ComfyUI startup log for the traceback (it names the package that failed), reinstalling requirements against ComfyUI's own Python, and using Manager's Try Fix. If this node is missing, every LayerStyle node is, and that shared import failure is the real problem.