VLM Track Detections

A detector sees a person in frame 1, then frame 2, then frame 3. Are they the same person? The detector doesn't know and doesn't care - it just returns boxes. VLMTrackDetections is the node that answers the "same?" question: it takes a sequence of per-frame detections and stitches them into tracks with durable object IDs, using a ByteTrack-style two-stage association. It's the standard tracking-by-detection pattern, packaged as a single node with no separate model.

You reach for it the moment your pipeline needs to talk about objects over time instead of boxes per frame: counting unique people in a clip, tracking one object while it moves, feeding a temporal reasoner, or building a persistent scene state. Detection tells you what is there; this node tells you that the thing in frame 12 is the same thing that was in frame 3.

How it works

The mechanism is ByteTrack-style: every frame's detections are matched against existing tracks in two passes - high-confidence detections first (strict matching), then low-confidence ones against whatever's left. Matching is IoU-based on the boxes, and when label_aware is on (default), detections are only matched to tracks of the same label, so a cat doesn't steal a person's track.

The controls that actually matter:

• high_threshold / low_threshold (0.6 / 0.1) - the confidence ladder. Above high, detections start tracks eagerly; between the two, they only extend existing tracks.
• match_iou_threshold / low_match_iou_threshold (0.3 / 0.2) - how much box overlap counts as "the same object."
• max_age_seconds (1.0) - how long a track can go unseen before it dies. Short = tracks break when the object is briefly occluded; long = tracks survive gaps.
• min_hits (2) - how many detections a track must accumulate before it's confirmed and emitted as a real track. Filters out one-frame false positives.
• emit_predictions (on) - whether to emit predicted boxes during gaps, which lets tracks survive short missed detections by extrapolating motion.
• fps_fallback (30) - used to convert frame counts to the seconds the age limits expect when the detection payload doesn't carry FPS.

Output is a single tracks (VLM_TRACKS) - durable IDs, ordered observations per track, ready for VLMTrackReport, VLMTrackAwareCrops, or VLMBuildSceneState.

The durability caveat to internalize

Track IDs are durable within the supplied sequence - and only within it. Different ComfyUI queue runs, or independently sliced chunks of video, are separate tracking sessions with fresh IDs. The pack is explicit that it won't pretend IDs are globally stable across separate queues. If you split a long video into slices, that's expected: track each slice, keep the mapping in your caller.

Install

Part of ComfyUI VLM Nodes (gokayfem/ComfyUI_VLM_nodes). ComfyUI Manager → search "VLM Nodes", or:

cd ComfyUI/custom_nodes
git clone https://github.com/gokayfem/ComfyUI_VLM_nodes
python -m pip install -r ComfyUI/custom_nodes/ComfyUI_VLM_nodes/requirements.txt

Run pip with ComfyUI's Python; the repo won't install its own torch. This node is pure association logic - no model download, works on any backend.

Gotchas

Thresholds are a tradeoff, not constants. Cranky low-res or fast-moving footage will need max_age_seconds bumped and min_hits lowered or you'll see tracks flicker; a scene with many similar objects will need label_aware and tighter IoU thresholds or tracks will swap. And remember fps_fallback only matters when the detections don't carry FPS - if your detector knows the real FPS, feed it properly and the seconds-based limits behave.