SAMURAI Refine

This is the workhorse of the pack, the one that does the actual video object segmentation. The two input nodes just let you point at an object in the first frame - SAMURAI Refine takes your box or your points and carries a mask of that object through every frame of the clip. That's the hard part, and it's the whole reason SAMURAI exists.

SAMURAI (the research repo this wraps, yangchris11/samurai, from an ECCV 2024 paper) is a zero-shot video object tracker built on SAM 2. SAM2 already propagates masks across video; SAMURAI adds a motion-aware memory that re-scores candidate tracks using optical-flow cues, which is what keeps the mask from drifting onto the background or jumping to the wrong object in cluttered scenes. If you've done rotoscoping by hand or keyframed masks frame by frame, this is the "just track it" button - feed it the first-frame prompt and it runs the rest.

How it works

On run, Refine loads a SAM2 video predictor via build_sam2_video_predictor from the vendored SAMURAI folder, writes your frames to a temp directory as JPEGs, initializes an inference state, drops your box or points in as the prompt on frame 0, and runs propagate_in_video. Each frame's mask logits get thresholded and stacked into one output tensor. It's a thin wrapper, not a reimplementation - the hard lifting happens in the code you were told to copy into the pack folder.

Inputs and outputs

The four required inputs:

• image (required) - video frames as an IMAGE batch, from a Load Video node.
• model_name - one of four SAM 2.1 checkpoints: sam2.1_hiera_small, base, base_plus, or large. Default is base_plus, and that's the one to reach for: the sweet spot of quality and speed for most footage. large is heavier, small is fast but drifts more.
• resolution (INT, default 1024, 64–2048) - the longest side frames get scaled to before inference. Bigger means more accurate, slower, and more VRAM.
• iou_threshold (FLOAT, default 0.1) - despite the name, in this pack it's used to binarize the mask: anything above it becomes 1. Leave the default; it's not something to tune.

Optional inputs: box (from Box Input), points and labels (from Points Input), and start_frame. Wire either the box path or the points path - both prompt the same object ID, so feeding both at once is asking for trouble. The single output is MASK, and it comes out shaped like [frames, objects, 1, H, W] - an extra object axis that most ComfyUI mask nodes don't expect. Plan on squeezing/reshaping it down to the standard per-frame [B, H, W] format before you composite it onto anything.

Installing it

The whole pack installs with one recipe, and it's not the usual clone-and-go:

cd ComfyUI/custom_nodes
git clone https://github.com/takemetosiberia/ComfyUI-SAMURAI--SAM2-.git samurai_nodes

Then copy your SAMURAI installation folder into samurai_nodes/, put the sam2.1_*.pt weights in samurai/sam2/checkpoints/, and pip install hydra-core omegaconf loguru. CUDA GPU required, Python 3.10+, and the README pushes conda hard - use the same environment for ComfyUI and SAMURAI. Nothing is auto-downloaded: no checkpoints, no SAMURAI code. It's all on you, which is the single biggest source of "why won't this load" posts.

Common issues

• Missing checkpoints. Refine looks for weights in samurai/sam2/checkpoints/; if they're not there you get a load error. The README's own troubleshooting section points straight at this.
• CUDA-only. The original SAMURAI code is locked to CUDA, so no CPU and no Apple Silicon. On either, this pack is a dead end.
• Prompt/resolution mismatch. The input nodes draw at full resolution; if resolution downscales your clip, the box or points aren't rescaled to match, so tracking starts off-target. Keep resolution at or above your video's longest side.
• Slow first run. It loads the checkpoint, writes frames to disk, then propagates - give it a minute before you assume it's hung. Long clips eat VRAM and time; the small checkpoint exists for a reason.

Honest verdict: this is a thin, single-commit wrapper around a genuinely good model - exactly the port people were asking for in late 2024, and then mostly didn't follow up on once it appeared. It works, it's fiddly, and for masked video-to-video or rotoscoping it's still one of the few free ways to get motion-aware tracking inside ComfyUI.