VLM Adaptive Frame Sampler

Ask a vision-language model to "watch" a ten-minute video and it has the same problem you do: most of the footage is dead air. The cheap fix is uniform sampling - grab every Nth frame - and that's exactly what wastes your tokens, because it keeps every boring frame and can still miss the two seconds where something actually happened. VLM Adaptive Frame Sampler is the smart version: it looks at the video's content, picks the frames worth showing the model, and stops there.

It's the middle link in the pack's recommended pipeline: Video Slice → this node → VLM Image Pixel Budget → any VLM. The logic is deliberately model-agnostic, so it saves you work no matter whether the VLM runs on CUDA, ROCm, Metal, or a hosted API.

How it works

The node computes a small thumbnail per frame (that's the thumbnail_size input, default 96px) and measures two cheap signals: scene changes and motion between frames. If you also feed it VLM_TRACKS from a tracking node, it adds a third signal - "this frame matters because a tracked object changed." It then picks frames that maximize coverage while respecting minimum_gap_seconds (so you don't get a burst of near-identical frames) and a hard budget of max_frames.

The important detail: it always preserves the real source frame index and timestamp. The selection output is a VLM_VIDEO_SELECTION socket that maps each sampled image back to its exact position in the original video - which matters a lot when a downstream VLM tells you "this happened at frame 7" and you need to know that means second 42.7 of the source, not the 7th frame of the batch.

Inputs and outputs that matter

• frames (IMAGE) - your full frame batch, straight from a video loader.
• fps - defaults to 24; get it right, because timestamps depend on it.
• max_frames - the budget. This is the knob you'll actually tune.
• strategy - the default Hybrid: scene + motion + tracks is what you want for most work; the four alternatives (Uniform coverage, Motion priority, Scene-change priority, Track-change priority) exist for deterministic experiments.
• tracks (optional VLM_TRACKS) - wire it in if you have tracking data; the hybrid strategy gets meaningfully better.

Outputs: sampled_frames (the IMAGE you send to your VLM), selection (the VLM_VIDEO_SELECTION socket), plus selection_json and diagnostics_json when you want to see exactly what it picked and why.

Installing this pack

This is one node in gokayfem's ComfyUI_VLM_nodes pack. Install via ComfyUI Manager (search VLM_nodes) or:

cd ComfyUI/custom_nodes
git clone https://github.com/gokayfem/ComfyUI_VLM_nodes
cd ComfyUI_VLM_nodes
python -m pip install -r requirements.txt

Run pip with ComfyUI's Python. No model downloads for this node - it's pure PyTorch frame analysis, so it works on the very first run.

Common issues

Two things catch people. First, fps mismatched with reality produces timestamps that drift from the source video - connect the fps output of GetVideoComponents when your frames came from a video, exactly like you would for the detection nodes. Second, a too-tight minimum_gap_seconds with a big max_frames can still flood the model with near-duplicates; raise the gap instead of just trimming the budget. On the pack's own test clip (60 frames, 1280×720) the sampler pulled 10 frames in about 0.44s and cut the frame×pixel analysis workload by over 11× before inference even started - that's the ballpark of what this node is for.