Video Prompt Reconstructor — Vision LLM

The job that nobody wants to do by hand: you're re-rendering or enhancing a long video in batches, and each batch needs a prompt that actually matches what's in those frames - not the prompt for batch one pasted onto batch twelve where the subject has walked across the room and changed jackets. Video Prompt Reconstructor solves it by looking. It grabs a few representative frames from your current video batch, sends them to a vision model, and returns a concise, visually-grounded prompt describing exactly what's on screen.

It's from the VRGameDevGirl pack, and it exists to feed the same batching machinery as the pack's video-enhance and builder pipelines: keep the description honest to the footage, and continuity between batches stops being a prayer.

How it works

The node does five things, in order:

1. Picks frames. frames_to_extract selects N frames spread evenly through the batch - 3 means first, middle and last. More frames = better coverage, more tokens spent.
2. Downscales and encodes them to JPEG data URLs (max 768px on the long side) so they can travel as regular image content to any OpenAI-compatible chat-completions endpoint.
3. Adds continuity context. If you feed overlap_frames (frames from the end of the previous batch), it prepends the last 2–3 of them so the model keeps the same subject, clothing and environment across the batch boundary.
4. Asks a vision model with a tight system instruction: describe only what's visibly confirmed - subject identity, pose, clothing, environment, lighting, palette - and explicitly never invent camera movement, lens choices, story, emotion, sound or dialogue. The output is stripped of markdown fences and "Here is your prompt:" chatter before it reaches you.
5. Caches. Prompts are cached on disk keyed by video filename + batch index + model + frame settings, so re-running a batch doesn't burn another API call (or another pass through your local server).

Inputs that matter

• video - the current batch as an IMAGE tensor.
• batch_index - which batch this is (0, 1, 2…). It keeps cached prompts separate per batch, so don't leave it at 0 for every chunk.
• llm_runner - pick your backend: LM Studio (local, default URL http://127.0.0.1:1234/v1/chat/completions), LLM API (cloud providers), or Custom Server (any OpenAI-compatible server, e.g. Ollama).
• llm_provider / model_name / api_url / api_key - provider (openai/anthropic/google/openrouter/grok) only matters on the API runner; api_key is a password field, blank for local LM Studio.
• max_prompt_length - 1200 default; raise it if you want more detail in the description.
• video_filename - optional, but it's what makes caching stable. Give it the source filename and identical re-runs skip the API entirely.

The single output, prompt, is a STRING that feeds your conditioning node.

Local vs. cloud - actually think about this one

The KB on LLM-in-the-graph has a deservedly stern warning attached: LLM-vision custom nodes are exactly the shape of thing that has been weaponized once (the ComfyUI_LLMVISION malware incident), so only run nodes like this from a pack you trust and keep api_key out of shared workflow JSON. The practical second point is that "send every batch of my video to a cloud vision model" is a privacy and cost decision you should make on purpose. This node is built for LM Studio - a local model keeps your frames off someone else's server and costs nothing per call, at the price of needing a vision model running on a second port while your GPU renders. qwen2.5-vl-7b is in the model list for exactly that reason.

Install

cd ComfyUI/custom_nodes
git clone https://github.com/vrgamegirl19/comfyui-vrgamedevgirl.git

or via ComfyUI Manager (search "VRGameDevGirl"). Restart, hard-refresh. The usual heavy pack requirements apply - and note this node specifically leans on the pack's Video Builder LLM runners, so don't prune files to "save space" if you want it to work.

Gotchas

• No frames → error. It needs a non-empty batch; feed it the actual batch, not a preview.
• Cache misses look like slowness, not bugs. If you change model or frame count, the cache key changes and you pay a fresh call. That's by design.
• overlap_frame_count over 6 is refused - the input caps at 6; 2–3 is genuinely enough for continuity. More just dilutes the current batch's frames.
• A vision model that can't see well (small local models on busy scenes) returns plausible but wrong prompts - spot-check batch 0 before trusting batch 40.