LayerMask: YoloV8 Detect(Advance)

Don't confuse this with ObjectDetectorYOLO8 elsewhere in this pack - they sound almost identical but do different jobs. That one outputs bounding boxes (BBOXES) for you to hand to another masking node. This one goes straight to masks: point it at an image, and depending on which model you picked, it either draws face/hand bounding-box masks or, with a segmentation model, traces the actual outline of what it detected.

The mechanism is standard YOLOv8 inference with one important branch: whether your chosen model has "seg" in its name. Non-seg models can only give you box-shaped masks - a rectangle around whatever was detected. Seg (segmentation) models trace the actual pixel outline, which is what you want if you need a mask that hugs the subject rather than a crude rectangle.

The inputs that matter:

• image - your source.
• yolo_model - a dropdown populated from whatever model files you've placed in the models folder; the README notes seg-suffixed models unlock outline masks, non-seg models only give boxes.
• mask_merge - all combines every detected mask into one, or pick a number (1 through however many detections you've got) to keep only the top N by confidence, merged.

Past those two required fields, there's a set of optional tuning knobs that follow standard Ultralytics YOLO conventions: conf (default 0.25) is the confidence threshold below which a detection gets dropped; iou (default 0.45) controls how aggressively overlapping boxes get merged via non-max suppression; classes lets you restrict detection to specific class names if your model supports multiple classes; device (default "auto") picks CPU vs GPU; max_det (default 300) caps how many detections get returned per image; retina_masks (default true) gives higher-resolution segmentation masks at some speed cost; and agnostic_nms (default false) merges overlapping boxes across different classes instead of only within the same class.

Outputs: mask (the merged result per your mask_merge setting), yolo_plot_image (a visual preview with detections drawn on the source image - useful for checking what actually got caught before trusting the mask), and yolo_masks - every individual detected mask, unmerged, if you need to handle them separately rather than as one combined blob.

Installing it: ComfyUI Manager, search "ComfyUI Layer Style Advance," or git clone https://github.com/chflame163/ComfyUI_LayerStyle_Advance into custom_nodes, then install_requirements.bat (or Aki variant) / pip install -r requirements.txt, then repair_dependency.bat, restart. Model files come from GoogleDrive or BaiduNetdisk (linked in the README) and go into ComfyUI/models/yolo. If you want outline masks rather than box masks, make sure the specific checkpoint you grab has "seg" in its filename - a plain detection-only model in that same folder will still show up in the dropdown, it just won't be able to give you anything but rectangular masks.

Where this trips people up: if masks come back as crude rectangles when you expected a tight outline, that's almost always a non-seg model in the dropdown, not a settings problem - swap to a seg-suffixed checkpoint. And this node is closely related to the whole "detect-then-mask" pattern that YOLO-based tools use across the ecosystem (the same idea behind ADetailer's face/hand fixing in other UIs) - if detections feel too permissive or too strict, conf is the first knob to reach for before anything else.