SAM2Matting Video Background (Streaming)

If you've ever matted a video the obvious way - load every frame into a batch, run per-frame background removal, composite - you know the wall you hit around frame 500: the RAM climbs, the swap file weeps, and ComfyUI gets closer to falling over with every frame you add. This is the node that sidesteps all of it. It's the star of the ComfyUI-SAM2Matting pack, and the README's recommendation whenever your end result is a normal video over one solid background color.

Why it doesn't blow up

Instead of holding the clip in memory, it runs a three-stage pipeline where frames and mattes live on disk. First it decodes, resizes, and caches model-resolution frames to temp storage (lossless Zstandard by default, CPU workers overlapping the work). Then it tracks sequentially on the GPU, writing each finished alpha as a PNG to disk, and - this is the clever bit - it prunes the SAM predictor's temporal state to its active attention window as it goes. So tracking-state memory plateaus instead of growing with clip length, which is the difference between a 10-minute render and a crash on a 2-minute clip. Finally it stabilizes each matte against its neighbors, decodes the source again, composites over your color in a bounded worker queue, and encodes the result.

The trade is temporary disk I/O and a second video decode for predictable host memory. Temp disk grows with clip length and image content, but RAM stays sane.

The inputs that matter

• video - a native VIDEO, straight from Load Video Path (Native) or core Load Video. Don't put Get Video Components in front of it; it wants the file-backed native type.
• initial_mask - one white-foreground seed mask on one frame. Rough is fine.
• mask_frame - the zero-based frame matching your mask. Tracking propagates forward, and backward from there when it needs to.
• mask_threshold - binarizes only the seed; the soft output matte is untouched.
• background_color - six-digit RGB hex, default #808080. That grey is a good neutral; any hex works.
• state_device - gpu (default) keeps the bounded tracker state on the GPU and out of system RAM; cpu trades VRAM for host RAM if you're tight on the former.
• streaming_options - optional connection from SAM2Matting Streaming Options. Leave it disconnected to use the recommended defaults.

What comes out

Two file-backed native videos, both ready for Save Video:

• video - your foreground composited over the color, H.264 MP4. Audio is preserved by default.
• matte_video - a lossless white-on-black matte (FFV1 in a Matroska container), with soft grey edges where the alpha is partial. No audio. This is the deliverable you'd pull into an editor if you want to re-composite elsewhere.

Neither output ever constructs a full IMAGE or MASK batch, which is the whole point.

The quick start

Open a clip with Load Video Path (Native), paint one black-and-white seed mask (white = subject), load sam2.1_base_plus with the model loader, connect everything here, set background_color to taste, and wire video to Save Video. The pack ships sam2matting_video_default.json as an example workflow - drag it onto ComfyUI to see the whole thing wired.

Where people get burned

The pack is new enough that most of the pain is environmental. It needs a recent ComfyUI with native VIDEO support; the checkpoint downloads automatically on first run to models/sam2matting/, so the first execution takes a while; and the SAM3 variant is slow (~9 FPS) and needs CUDA plus a big RAM box. If matte edges flicker on fast or translucent detail, that's the edge_stabilization control in the streaming options node - leave it at its gentle default unless you have a reason to change it. And remember the upstream license is non-commercial CC BY-NC-SA, which matters if you're thinking of charging for renders.