LayerMask: MaskEdgeUltraDetail V2

Your background remover found the subject but butchered the edges. The hair looks like it was cut out with kitchen scissors, the fur is a hard jagged line, and the moment you drop the cutout on a colored background it screams "composited." That's the exact wound this node exists to close. You feed it your rough mask plus the original image, and it rebuilds the edge - the messy transition zone between subject and background - as proper fractional alpha instead of a hard on/off line.

This is a refinement node, not a segmentation node. It doesn't find the subject. Something upstream - BiRefNet, SAM, a Segformer node, RMBG, whatever - hands it a mask, and MaskEdgeUltraDetail makes that mask's border actually usable. If your cutout survives a grey preview but falls apart against a red wall, that's a segmentation mask being asked to do a matting job, and this is the fix.

How it works

Under the hood it's trimap matting. It takes your incoming mask, erodes it inward and dilates it outward to carve a band of "unknown" pixels around the edge, then runs a matting algorithm across only that band to solve for how much of each pixel is really foreground. That's why the two erode/dilate knobs matter more than anything else - they set how wide the "figure this out" zone is.

The inputs and outputs that matter

• method - your matting engine. VITMatte is the quality option (a transformer matting model; downloads on first use), PyMatting and GuidedFilter are lighter and need no model download. Start with VITMatte if you can, fall back to the others if it's too slow or won't download.
• edge_erode and edte_dilate (yes, "edte" is a typo baked into the node) - the trimap band. Wider = the matting looks at more pixels around the edge, which catches wispy hair but can get mushy. This is the pair you'll actually tune.
• black_point / white_point - levels on the resulting alpha. Nudge these to kill haze or firm up a soft edge.
• max_megapixels (default 2) - an internal downscale cap. VITMatte is hungry; if you OOM or it crawls, this is your throttle.
• device - cuda or cpu, in case VRAM is tight.

It's the V2, so it takes an image plus a mask and returns both a cleaned image and a refined mask. Wire the mask into wherever you needed a clean cutout; wire the image if you want the matted RGB.

How to install it

Grab the whole pack. In ComfyUI Manager, search LayerStyle, install, restart. Or manually:

cd ComfyUI/custom_nodes
git clone https://github.com/chflame163/ComfyUI_LayerStyle
pip install -r ComfyUI_LayerStyle/requirements.txt

then restart. It's a big pack with a heavy dependency list, so give the pip step a minute. The node lands under the 😺dzNodes → LayerMask menu.

Common issues & troubleshooting

It hangs at "Downloading hustvl/vitmatte-…". The VITMatte method pulls a model from HuggingFace the first time you run it. If you're behind a firewall or HF is blocked, it just sits there - this is the single most-reported snag with the Ultra nodes. Either sort out network access and let it finish once, or switch method to PyMatting or GuidedFilter, which need no download.

GuidedFilter errors out. That method needs opencv-contrib-python (the ximgproc module). If another pack installed plain opencv-python over it, GuidedFilter breaks - reinstall the contrib build or just use a different method.

The whole pack shows "import failed" in Manager. LayerStyle pulls a large transformers dependency, and the most common LayerStyle failure by far is a broken transformers install (often a stray TensorFlow in the environment) taking the entire pack down on import - so none of its nodes appear. Read the traceback for the offending library, "Try Fix" in Manager, or repair that one dependency.

It's slow. VITMatte at full res is the cost. Drop max_megapixels, or accept PyMatting/GuidedFilter for a speed-for-quality trade.