VLM Video Temporal Reasoner

Most video-VLM workflows fail for the same reason: they feed a model 300 full-res frames and ask for a summary, and the model drowns. VLMVideoTemporalReasoner is the pack's answer to that - a single node that does the whole pipeline itself. It adaptively samples your video down to a few meaningful frames, downsizes them for analysis, runs a real vision-language model, parses the response into validated timestamped events, and hands you summary, events, sampled frames, and diagnostics. No assembly required.

This is the "I want answers about a video without building a five-node chain" node. It's the pack's flagship video-intelligence path, and it's genuinely good at the job: on the pack's own test clip (157 frames, RTX 3090), hybrid sampling picked 12 frames and cut temporal input by over 90% before the model ever ran.

How it works

Four stages, all inside one node:

1. Adaptive sampling - it picks frames that matter: scene changes, motion, track changes, plus uniform coverage for safety. The strategy input lets you bias it (hybrid is the default; motion-priority and scene-change-priority are there for deterministic experiments). max_frames caps the pick, minimum_gap_seconds stops it from stacking near-identical frames.
2. Downscale the analysis copy - analysis_max_side (default 448) shrinks only the frames the VLM sees, keeping tokens low. Zero keeps source resolution.
3. Inference - it runs one of 12 curated local models (default SmolVLM2 2.2B Video; pick Qwen 3 VL 2B for a fast generalist or a larger Qwen for quality). memory_mode switches between ComfyUI-managed BF16, 4-bit/8-bit bitsandbytes, and CPU. Models download on first run into ComfyUI/models/LLavacheckpoints - Gemma 3 requires accepting its license on Hugging Face, which is why it's not the default.
4. Strict parsing - the response becomes VLM_EVENTS, validated against the timestamp/evidence contract. Small models often return "evidence" as positions in the image batch they were given; the parser accepts that only when unambiguous and maps it back to real source indices. Malformed JSON, out-of-range timestamps, bad confidence, duplicate evidence - all rejected when strict_output is on (default).

Inputs worth touching

• frames + fps - your video batch (use core GetVideoComponents / Video Slice upstream) and its frame rate. Timestamps are wrong without a real fps; don't leave it at 24 if your clip is 30.
• task / question - action timeline, robotics scene understanding, safety/anomaly monitor, detailed summary, or a free question.
• max_frames / max_events / max_new_tokens - your budget controls: how many frames to sample, how many events to allow, how long an answer to let the model write.
• model / memory_mode - the quality/VRAM dial. 2B-class models on a 12GB card are the comfortable default; reach for 4-bit quantization or the 450M edge models before you reach for a bigger GPU.
• tracks / scene_state (optional) - feed in VLM_TRACKS or a scene state and the reasoner grounds its reasoning in what's already known.

Outputs

• summary (STRING) - the human answer.
• events (VLM_EVENTS) - validated, ordered events with timestamps and evidence; the typed payload for VLMBuildSceneState or custom automation.
• selection (VLM_VIDEO_SELECTION) - the exact sampled-frame mapping.
• sampled_frames (IMAGE) - see what the model actually saw.
• raw_response, diagnostics_json, events_json, selection_json - the raw model output and machine-readable versions of everything.

That's the killer feature honestly: you get the pretty answer and the receipts, without re-running anything.

Install

Part of ComfyUI VLM Nodes (gokayfem/ComfyUI_VLM_nodes). ComfyUI Manager → search "VLM Nodes", or:

cd ComfyUI/custom_nodes
git clone https://github.com/gokayfem/ComfyUI_VLM_nodes
python -m pip install -r ComfyUI/custom_nodes/ComfyUI_VLM_nodes/requirements.txt

Run pip with ComfyUI's Python, and never let the repo install or replace torch. First execution downloads your chosen model (respects HF_TOKEN); quantized modes pull official bitsandbytes wheels automatically on supported platforms.

Gotchas

unload_after defaults to false, which caches the model per node instance for fast re-runs - flip it on if a second big model must run right after. enable_thinking stays off for latency; turn it on when a task genuinely needs reasoning before answering. And the biggest one: don't fight the validation. If strict_output errors, it's usually telling you the model went off-contract - switch to a structured-output-capable model (the Qwen 3 VL family is the reliable one) rather than weakening validation to accept garbage.