✨Video Caption (VLM)

Found a clip you want to recreate, but the person who made it didn't share a prompt? ✨Video Caption is the video cousin of image captioning: it shows a vision-language model a handful of frames from your video and asks "what's happening here?" The result is a text prompt you can feed back into a video model. It's part of the ✨Prompt Assistant pack, so same caveat as the rest of it - this is an API client, not a local model. No weights to download.

The auto-captioning trick is a well-worn pattern in ComfyUI by now - people have been chaining Ollama vision models onto LTX and video pipelines for a while. What this node does differently is package it: frame sampling, model limits, and video-type compatibility are all handled for you instead of hand-wired.

How it works

You feed it frames one of two ways: an IMAGE batch into video_frames (works with almost any frame-sequence node), or a native VIDEO type into the video input - the tooltip says VHS, and the source specifically handles VHS-style dicts and ComfyUI's own VideoInput objects. Give it either one; it doesn't need both. The pack itself doesn't load video files, so drive it from a video loader node.

Then it samples frames. Auto (Uniform) splits the clip into equal intervals and takes the midpoint of each - frame_count frames, 1 to 32, default 5. Manual (Indices) lets you type exact frames like 0,10,20 or ranges like 0-10, with negative indices counting from the end. Selected frames get JPEG-encoded and downscaled on the fly (fewer frames → higher resolution sent to the model), then sent to the VLM along with a system rule and your optional user_prompt.

The node is smart about model limits: it checks the model's max supported images - Gemini-class models get 3000, Qwen-class get 100, unknown models a safe 10 - and truncates the frame list if you overshoot, warning you in the UI instead of hard-failing. That was an actual bug fix in V2.0.6, when some models were misjudged and crashed mid-run.

The inputs and outputs that matter

• video_frames or video - connect one of these, or the node errors out.
• frame_count - 5 is a solid default. More frames = richer caption but more tokens per call.
• sampling_mode / manual_indices - manual mode is your friend when a specific moment in the clip is the point (a single action beat, a cut).
• rule - the default 视频复刻与重构 is tuned to write Wan 2.1/2.2-format video prompts, which is exactly the model family the community runs this against. There's also a storyboard-deconstruction preset.
• vlm_service - 智谱/glm-4.6V-Flash by default; swap to Ollama for a fully local run.

Outputs are caption_text (the prompt string - wire it into your video model's text encode) and preview_frames (the exact frames it sampled, as an IMAGE batch). That preview output is a debugging gift: check it once and you'll instantly see whether your sampling captured the moment you cared about.

Install

Manager → search Prompt Assistant, install, restart. Manual route:

cd ComfyUI/custom_nodes
git clone https://github.com/yawiii/ComfyUI-Prompt-Assistant.git

Restart, then set an API key in the pack's config (Zhipu free tier is fine) or point it at Ollama. The pack's dependencies are light - httpx, imageio, imageio-ffmpeg.

Where people get burned

The big one is feeding it too many frames. frame_count caps at 32, but your VLM's context is the real ceiling - crank it up and you'll either burn tokens or hit the truncation path. Manual indices that don't parse (0-10,50 style typos) fall back to a default 8-frame uniform sample rather than erroring, which is nice but can silently give you the wrong frames - check preview_frames. And remember every caption is a single API round-trip, so a long clip means a slow queue. For quick iterations, 5 frames is usually plenty.