Nodes/CoachBate/CoachBate H3 Segm Mask (YOLO)
ComfyUI Node

CoachBate H3 Segm Mask (YOLO)

The difference between a pasted rectangle and a pasted subject

By CoachBate·Created 3 months ago·Updated 22 days ago· 2
CoachBate H3 Segm Mask (YOLO)
  • crops
  • transform
  • refined_crops
  • masks
  • report
◄model▾►
◄confidence0.30►
◄instancebest_overlap►
◄dilation0►
◄temporal_smooth5►
◄mask_sourceunion►
◄detect_scales1.0, 0.7, 0.5, 0.35, 0.25►
◄fallback_shapenone►

Silhouette or rectangle

This is the same battle that was fought over face detailing years ago, and it had the same answer. A bounding-box detector gives you a rectangle, so the patch you paste back is a rectangle, and its edges can show. A segmentation detector gives you a polygon that follows the subject, so the mask hugs the outline and the seam mostly disappears. That's the whole argument, and it's the whole job of CoachBate H3 Segm Mask (YOLO).

It runs a YOLO segmentation model over the crops from CoachBate H3 Subject Track + Crop and returns temporally smoothed masks for the stitch's masks input. Without it, the stitch pastes the tracked box (or an ellipse inside it). With it, the paste falls on the actual silhouette.

The multi-scale trick, which is the interesting bit

You'd think you'd just run the model on the crop. You'd be wrong about half the time. A segmentation model trained on medium shots scores a subject that fills half the crop near zero - the framing is outside what it was trained on. Shrink that same crop to half size and it scores 0.7–0.8. So this node detects at several scales (detect_scales, default 1.0, 0.7, 0.5, 0.35, 0.25, comma separated, 0.1–1.0) and keeps the best score. If your masks are coming back empty on a big close-up subject, that default list is the reason it usually still works.

The rest of the mechanism is defensive. instance picks which detection to keep when the model finds several in a crop - best_overlap (the one whose box overlaps the tracked rect most, i.e. the subject the tracker is actually following), union (all of them) or largest. On frames the model misses entirely, fallback_shape decides what happens: none pastes nothing and those frames keep their original pixels, which is right when the subject genuinely isn't there yet; ellipse or rect uses the tracked box, for detectors that drop frames the subject is in. temporal_smooth (default 5) averages across the mask stack so the mask edge doesn't shimmer - set it to 1 and you'll see exactly what it was preventing.

Inputs and outputs

Required: crops, transform, model, confidence, instance, dilation, temporal_smooth, mask_source. Optional: detect_scales, fallback_shape, refined_crops.

  • crops - this is the input crops, not the decoded result. The mask has to describe where the subject is in the footage being replaced. That's the one input people get wrong.
  • model - a segmentation model from models/ultralytics/segm, e.g. CockAndBallYolo8x.pt for the NSFW case, face_yolov8m-seg_60.pt for faces. A bbox-only model produces no masks at all and every frame silently falls back to a rectangle.
  • mask_source - input traces the silhouette from the tracker's crops (FaceDetailer's rule: right when the subject keeps its shape). output traces the new silhouette from the refined crops, which is what you want when the model reshapes or enlarges the subject - mask from the input only and everything H3 added outside the old outline gets clipped away. union (the default) takes both, so neither the old edge nor the new one is left behind. output and union need refined_crops connected and quietly fall back to input if it isn't.
  • dilation - grows the mask in canvas pixels. Leave it at 0: the stitch has its own mask_dilation that does the same thing later, and doing it here means growing before temporal smoothing, which is rarely what you intended.

Outputs: masks (MASK - straight into CoachBate H3 Subject Stitch Back's optional masks) and report.

When you use a silhouette mask, dial the stitch's feather down - 2–8 source pixels is the range. A rectangle needs the blur; a polygon doesn't.

Install

cd ComfyUI/custom_nodes
git clone https://github.com/CoachBate/ComfyUI-CoachBate.git

ComfyUI Manager users can search "CoachBate" instead; restart ComfyUI afterwards. The node lands under CoachBate / H3. It needs ultralytics installed - that's the runtime, separate from the weights - and the .pt files in models/ultralytics/segm/. Those folders are registered by ComfyUI-Impact-Subpack; the node looks in them directly if the Subpack isn't present. Note the licensing position while you're here: Ultralytics is AGPL-3.0 and it has been the vector for at least one real supply-chain incident, so pin your version rather than taking whatever ships latest.

Where people get burned

  • Every frame falls back to a rectangle. You picked a bbox model, not a segm one.
  • The detector dropdown is short or empty. Weights aren't in models/ultralytics/segm/, or ultralytics isn't installed. The error names the folders it searched.
  • Masks vanish after H3 enlarged the subject. mask_source = input with the refined crops wired but unused as a source - switch to union.
  • Shimmering mask edges. temporal_smooth is at 1. Turn it back up.
CategoryCoachBate/H3

Inputs (11)

NameTypeDefaultDescription
cropsIMAGEThe 'crops' output of H3 Subject Track + Crop - the INPUT crops, not the decoded result. The mask must describe where the subject is in the footage being replaced.
transformH3FACEXFORMFrom H3 Subject Track + Crop. Supplies the tracked rect per frame, used to pick the right instance and as the fallback mask on frames the model misses.
modelCOMBOA YOLO *segmentation* model from models/ultralytics/segm. A bbox-only model produces no masks and every frame falls back to the rect.
confidenceFLOAT0.300.05–0.95—
instanceCOMBObest_overlapWhich detection to keep when the model finds several in a crop. best_overlap: the one whose box overlaps the tracked rect most (the subject the tracker is following). union: all of them. largest: biggest mask area.
dilationINT00–128Grow the mask here, in canvas px. H3 Subject Stitch Back has its own mask_dilation that does the same thing, so leave this at 0 and use that one unless you need growth before temporal smoothing.
temporal_smoothINT51–31Frames of averaging across the mask stack. 1 disables it and the mask edge will shimmer.
mask_sourceCOMBOunionWhich crops the silhouette is traced from. input: the tracker's crops (FaceDetailer's rule - right when the subject keeps its shape). output: the refined crops H3 returned - the NEW silhouette, for a subject the model reshapes or enlarges; with input masking everything it added outside the old outline is cut off. union: both, so neither the old nor the new edge is left behind. output/union need refined_crops connected and fall back to input if it is not.
detect_scalesoptSTRING1.0, 0.7, 0.5, 0.35, 0.25Scales the crop is detected at, best score wins. A segm model trained on medium shots scores a subject filling half the crop near zero; the same crop at half size scores 0.7-0.8. Comma separated, 0.1-1.0.
fallback_shapeoptCOMBOnoneWhat to paste on frames the model finds no silhouette on (input or refined crop). none: nothing - those frames keep their original pixels, which is right when the subject is not actually there yet. ellipse / rect: the tracked box, for detectors that miss frames the subject IS in.
refined_cropsoptIMAGEThe decoded result of the H3 pass (VAEDecode output), same frame count as crops. Used by mask_source output / union.

Outputs (2)

NameTypeDescription
masksMASK—
reportSTRING—