VLM SAM2.1 Video Segmentation

Detectors find objects but they don't follow them. Run detection on every frame of a clip and the "person" on frame 1 is just a box that has no connection to the "person" on frame 90. VLM SAM2.1 Video Segmentation fixes exactly that gap: you seed it once - with detections, a single bounding box, or a mask - and Meta's SAM2.1 propagates that object's segmentation through the entire video batch, returning per-object tracks with stable IDs.

It's the segmentation-and-tracking core of gokayfem's VLM_nodes pack, built on SAM2.1 Hiera in four sizes: Tiny (fast), Small, Base+, and Large. The pack's video-memory strategy is explicit: start with Tiny, get the pipeline correct, then scale the model up.

How it works

You feed it images (the frame batch), pick a model, and seed it one of three ways: detections (a VLM_DETECTIONS socket - ideal from VLMOpenVocabularyDetection), a single core bounding_box, or a seed_mask. It reads the seed at seed_frame (default 0), then propagates segmentation frame by frame through the batch. Object IDs assigned at the seed frame stay stable for that video session - that's what makes the returned VLM_TRACKS genuinely useful for counting or identity work.

Outputs: tracks (the typed socket with durable IDs), json (the report), union_masks (all objects per frame), object_masks (per-object full-resolution masks, opt-in), and preview (an overlay batch). Connect fps from GetVideoComponents so every track observation gets the right timestamp.

The VRAM-conscious defaults are the good part: video frames and inference state stay on CPU (keep_video_on_cpu=true), the vision-feature cache is limited to one frame, and mask_output defaults to union_only because per-object full-resolution mask volumes are enormous. render_preview=false skips the extra full-size overlay copy on long clips, and unload_after=true frees the model for whatever runs next.

Installing this pack

Same one-pack install. ComfyUI Manager: search VLM_nodes. Or:

cd ComfyUI/custom_nodes
git clone https://github.com/gokayfem/ComfyUI_VLM_nodes
cd ComfyUI_VLM_nodes
python -m pip install -r requirements.txt

Run pip with ComfyUI's Python. First run downloads the chosen SAM2.1 weights (Tiny is smallest; Large is a real download) into ComfyUI's model directory, so give it time and an internet connection.

Common issues

Where people get burned: seeding from detection boxes that sit slightly off the object makes the first-frame propagation start with a bad prompt - keep box_threshold sensible upstream and consider VLMDetectionsToMasks for tighter seeds. mask_output=union_and_objects on a long 1080p clip is how you exhaust RAM in one click; keep union_only unless you genuinely need per-object volumes. And remember the session rule from the README: a Video Slice is an independent propagation session - IDs are durable within one session only, so for long media use bounded slices, reseed each one, and keep the mapping in the caller. The pack does not pretend IDs are globally stable across separate queues.