SAM2 Video Detector (SEGS)

SAM2 Video Detector (SEGS) finds a subject in a video and tracks it through every frame, producing a SEGS with a coherent mask per frame. The word doing the work is tracking. Instead of running a fresh detection on each frame and hoping the results line up, it locks onto the subject once and follows it, which is what SAM2 (Meta's video-capable Segment Anything) was built to do. For per-frame detailing on a clip, that temporal coherence is the difference between a clean fix and a flickering mess.

Contrast it with the naive approach. Simple Detector for Video runs the detector independently on each frame and stacks the masks - fine, but the masks jitter, the detector occasionally misses a frame, and the mask shape wobbles. SAM2's video tracking carries the segmentation forward with temporal awareness, so the mask stays stuck to the subject as it moves. That's why this node exists as a separate, better path for video: same SEGS output, steadier masks.

How it works

It's a two-model pipeline. A BBOX_DETECTOR finds the subject on a frame (bounding boxes with confidence), and a SAM2 model - loaded through SAMLoader with a SAM2 model selected - takes that detection and propagates a precise segment across the whole clip using its video-tracking machinery. The result is bundled as SEGS with a batch of per-frame masks, ready for the video detailer. SAM2 support landed in Impact Pack v8.18, so you need a recent build and a SAM2 model file, not the older SAM weights.

The inputs and outputs that matter

• image_frames (required) - the clip, as a batch of frames.
• bbox_detector (required) - a YOLO detector (from the Subpack) that finds the subject to track.
• sam2_model (required) - a SAM2 model, loaded via SAMLoader with a SAM2 model selected. This is the piece that does the tracking; a plain SAM model won't do it.
• bbox_threshold (default 0.5) - detection confidence cutoff for the initial find.
• sam2_threshold (default 0.5) - SAM2's confidence gate for the tracked segment.
• crop_factor (default 3) - context captured around the subject for the downstream detailer.
• drop_size (default 10) - ignore detections smaller than this.

Single output: SEGS - per-frame masks tracking the subject, feeding SEGSDetailer For Video and then SEGS Paste.

How to install it

Install the pack via ComfyUI Manager (search ComfyUI Impact Pack, install, restart), or manually:

cd ComfyUI/custom_nodes
git clone https://github.com/ltdrdata/ComfyUI-Impact-Pack comfyui-impact-pack
cd comfyui-impact-pack
python -m pip install -r requirements.txt   # ComfyUI's python; python_embeded on portable

then restart. Two model requirements beyond the base install: a SAM2 model (Impact Pack supports facebookresearch/sam2 models as of v8.18 - download one and select it in SAMLoader), and a YOLO detector from the Impact Subpack for the bbox_detector. Linux: libgl1-mesa-glx and libglib2.0-0 for OpenCV.

Common issues & troubleshooting

SAMLoader won't give me a SAM2 model. You need an actual SAM2 checkpoint, not the classic sam_vit_b weights that auto-download with the pack. Grab a SAM2 model and place it where SAMLoader looks, then select it - SAM2 tracking only works with a SAM2 model.

The track drifts off the subject mid-clip. SAM2 propagation can lose a fast-moving or occluded subject. Tighten the initial detection (raise bbox_threshold so it locks onto the right thing) and make sure the subject is clearly present in the first frames where tracking initializes.

It's heavy. Video tracking plus detailing is one of the more demanding things you can ask ComfyUI to do - budget RAM generously, and keep clips short while you dial in settings. If it's too much, Simple Detector for Video is the lighter (jitterier) fallback.

Empty SEGS. The bbox detector found nothing to hand SAM2. Check that your YOLO model matches the subject and lower bbox_threshold if the subject is small or stylized.