VLM Track-Aware Semantic Crops

Tracking gives you object IDs; it doesn't tell you what the objects are. That's a VLM's job - but feeding a VLM every full frame of a long video is slow, wasteful, and full of redundant information. VLMTrackAwareCrops is the middle step that makes it cheap: it picks a few representative crops per tracked object across the video, adds context around each box, and returns them letterboxed into one batch. The VLM then labels identities from a handful of crops instead of re-reading the whole clip.

The pattern the pack pushes for video identity: detect → track → this node → a VLM that names what each track is. You're trading "look at everything, always" for "look at the two or three moments per object that actually show what it is." For a 60-second clip, that's the difference between hundreds of full frames and a dozen 448px crops.

How it works

It takes your full frames (IMAGE) batch plus the tracks (VLM_TRACKS) from a tracker, and for each track picks representative observations spread across its lifetime - a few moments that cover how the object looks and moves. For each chosen observation it crops the box, then scales up the crop to include context_scale (default 1.35) worth of surrounding scene so the VLM has spatial context, not just a floating face. All crops get resized to output_size (default 448, the sweet spot for most VLMs) and letterboxed into one uniformly-sized batch.

The knobs:

• crops_per_track (3) - how many representative observations per track. More = better identity coverage, more tokens.
• max_crops (32) - the hard cap on total crops across all tracks. This is your budget control: keep the final batch small enough for the VLM's context window.
• output_size (448) - the target square size.
• context_scale (1.35) - how much scene context around each box. Raise it for small objects, lower it when crops start overlapping each other's subjects.

Outputs are crops (the letterboxed IMAGE batch) and crop_manifest_json - a manifest mapping each crop back to its track ID, frame index, and box, so a VLM's answer about "crop #7" can be traced to "track 3, frame 41." That mapping is the part that makes downstream labeling trustworthy.

Where it fits

Wire it after VLMTrackDetections, VLMSAM2VideoSegmentation, or the SAM3 adapter, and feed the crop batch into a VLM (Modern VLM, Moondream, Florence-2) with a prompt that asks it to label each crop. The manifest means you can map labels back to tracks and build per-object identity, which is the prerequisite for scene-state nodes like VLMBuildSceneState.

Install

Part of ComfyUI VLM Nodes (gokayfem/ComfyUI_VLM_nodes). ComfyUI Manager → search "VLM Nodes", or:

cd ComfyUI/custom_nodes
git clone https://github.com/gokayfem/ComfyUI_VLM_nodes
python -m pip install -r ComfyUI/custom_nodes/ComfyUI_VLM_nodes/requirements.txt

Run pip with ComfyUI's Python; the repo won't install its own torch. This node does no inference and downloads no models - the VLM it feeds is your choice.

Gotchas

The crop-identity loop is only as good as its tracking, so a tracker that swaps IDs mid-clip will hand you crops of two different objects on one track - check the manifest and VLMTrackReport before trusting labels. And don't over-crop: max_crops is your context budget, and 32 crops at 448px already eats a chunk of a small model's window. Fewer, well-spread crops beat more.