VLM Video Reasoning Prompt

Asking a VLM "what happened in this video?" is easy. Getting back an answer you can trust - with timestamps, frame evidence, and a consistent structure - is not. VLMVideoReasoningPrompt is the node that builds the other half of the deal: it takes your sampled video selection and produces the strict, structured prompt (plus its matching JSON schema) that turns "vague temporal summary" into "validated events with timestamps."

It's the companion to VLMVideoTemporalReasoner for people who want to run their own VLM - local or hosted, any model - instead of using the all-in-one node. The reasoner packages sample→prompt→infer→parse into one call; this node hands you the prompt and schema so you pick the model. Same contract either way, which is the point.

How it works

You feed it a selection (VLM_VIDEO_SELECTION) - the exact mapping from sampled frames to source frame indices and timestamps, produced by VLMAdaptiveFrameSampler or the reasoner's own sampling. It then builds a prompt that tells the model: here are the frames, here's the timestamp contract, return your answer as events with evidence in the specified form. Choose a task (action timeline, robotics scene understanding, safety/anomaly monitor, detailed temporal summary, or answer a question), optionally give it a question, and cap max_events.

The two outputs are:

• prompt (STRING) - the assembled reasoning prompt, including the frame/timestamp mapping the model needs to ground its answer.
• json_schema (STRING) - the JSON Schema for the event structure. This is the contract enforcer: feed it to a model that supports structured output, or use it to validate what a model returns.

The strictness matters because small VLMs have a notorious failure mode: they return "evidence" as positions in the image batch they were given rather than source frame indices. The pack's parsing layer (in VLMEventsFromVideoJSON, which pairs with this node) only accepts that form when every value is an unambiguous valid supplied-image position, then maps it back to the immutable source selection. Garbage evidence, out-of-range timestamps, malformed JSON - all rejected, not silently accepted.

Optional scene context

There's one optional input: scene_state (VLM_SCENE_STATE). If you've built a persistent scene state from tracks (via VLMBuildSceneState), feeding it in lets the model reason over "what I know about the scene" plus "what's new in these frames" - materially better answers on long clips where the model shouldn't have to re-derive obvious facts.

Install

Part of ComfyUI VLM Nodes (gokayfem/ComfyUI_VLM_nodes). ComfyUI Manager → search "VLM Nodes", or:

cd ComfyUI/custom_nodes
git clone https://github.com/gokayfem/ComfyUI_VLM_nodes
python -m pip install -r ComfyUI/custom_nodes/ComfyUI_VLM_nodes/requirements.txt

Run pip with ComfyUI's Python; the repo won't install its own torch. No model, no downloads here - it builds text only.

Gotchas

The prompt's quality is bounded by the sampling that precedes it. A selection with poorly-chosen frames yields events with nothing to ground on, no matter how strict the schema. And remember the schema is strict by design: if your chosen model ignores structured-output instructions and returns prose, VLMEventsFromVideoJSON will refuse it. That's the contract working - switch models or fix the prompt rather than weakening validation.