LayerMask: MaskEdgeShrink

You cut out a subject, composite it onto a new background, and there's a thin ugly halo of the old background clinging to the edge. This node fixes that. It shrinks the mask's edge inward so the fringe pixels get excluded, while trying to hold onto genuine edge detail so you don't just eat the subject's silhouette. It's the cleanup pass that turns a rough matte into something you can actually composite.

Edge halos are the single most common complaint with cutouts. As the background-removal folks put it, even good segmentation models "break the lines around characters," leaving a rim of contamination. Shrinking the edge a hair is the cheap, reliable fix when you don't want to re-run a whole matting model.

How it works

It erodes (contracts) the mask boundary by a controllable amount, softens the new edge, and includes a "reserve" control so it doesn't shrink so aggressively that it destroys fine protrusions. The edge_reserve value is the smart part - it decides how much of the original edge detail to keep versus how hard to pull in.

The inputs and outputs that matter

• mask (MASK) - the matte to tidy.
• invert_mask (BOOLEAN, default true) - heads up, this defaults to true. If your result comes out backwards, this is why. Many LayerStyle mask nodes treat white as the background, so the default inversion is intentional but bites first-timers.
• shrink_level (INT, 0–16, default 4) - how far the edge pulls in. This is the main dial. Start low (2–4); too much and you carve into the subject.
• soft (INT, 0–64, default 6) - feathering on the new edge, so the transition isn't a hard cut.
• edge_shrink (INT, default 1) and edge_reserve (INT, 0–100, default 25) - how much fine edge detail to preserve while shrinking. Raise edge_reserve if shrinking is chewing up hair or thin features.

Output is a single cleaned mask (MASK).

How to install it

ComfyUI Manager: search ComfyUI Layer Style, install, restart. Or:

cd ComfyUI/custom_nodes
git clone https://github.com/chflame163/ComfyUI_LayerStyle

Restart to install requirements. No model download.

Common issues & troubleshooting

The pack-wide gotcha is IMPORT FAILED at startup - that's the pack not loading, not this node. Across ComfyUI_LayerStyle it traces to a stale install from before the split into ComfyUI_LayerStyle and ComfyUI_LayerStyle_Advanced (delete, reinstall), or a broken dependency crashing the import. Reinstall clean via Manager.

The node-specific trap is invert_mask defaulting to true - the number-one "why is my mask inverted" moment with this whole family of nodes. If the shrink is happening on the wrong side, flip it. Beyond that: if the subject's getting eaten, your shrink_level is too high or edge_reserve too low. For genuinely hairy edges (fur, flyaways) this node alone won't recover detail - it removes fringe, it doesn't reconstruct. Pair it with MaskEdgeUltraDetail (the matting-based refiner in this same pack) when the edge is the whole problem.