Video Captioner

Video captioning is where image captioners run out of road, because a video is not a JPEG. Video Captioner is the WepeNerd node that handles that gap: it takes a VIDEO input plus the GGUF_LLM_CONFIG from Local AI Model, and returns a caption describing subjects, actions, camera motion, and how the scene develops over time. It's for building video LoRA training sets, or for turning an existing clip into the prompt for a video-to-video workflow without writing a sentence yourself.

The style dropdown mirrors the image node - Dataset, Detailed, Motion + Camera, Short, Custom - with an optional instruction override. Motion + Camera is the one you'll actually want for video: it tells the model to focus on temporal action and shot changes, which is what video training captions need and what a generic image-style description misses.

The mechanism is the interesting part, and it's capability-aware. A vision-language model can "see" video in two ways:

• Native video - the backend (recent llama.cpp builds with multimodal support) accepts a real input_video payload.
• Sampled frames - the node decodes the clip into a few chronological JPEG frames with timestamps and sends those.

Video Captioner runs in auto mode and doesn't guess: it queries llama-server's /props endpoint. If the backend explicitly advertises native video, it uses it; otherwise it falls back to timestamped sampled frames. Incomplete metadata is treated as unknown and defaults conservatively to sampling. For long file-backed clips, the sampling uses PyAV seek (not full decode into RAM), so memory scales with the number of frames you actually grab, not the clip's total length. Reasonable defaults: 12 samples at 2 fps, capped at 24 frames.

Two things to know before you commit to it. First, audio and dialogue are never inferred - the caption describes what's visible, full stop. Second, the whole thing needs the vision projector selected in Local AI Model; without an mmproj file the node errors out. Same projector-must-match-model warning as the image captioner.

The one output, captions, is a single string (not a per-batch list like the image version) - wire it into whatever node wants the text, or just read it from the preview. If you need finer control - forcing native video, choosing uniform vs fixed-fps sampling, or a banned_phrases cleanup list - that's Video Captioner (Advanced). This one is the set-and-forget version.

Install is the pack standard: ComfyUI Manager, search WepeNerd, or git clone https://github.com/WepeNerd/ComfyUI-WepeNerd.git into custom_nodes plus pip install -r requirements.txt. Remember the pack's requirements.txt includes av (PyAV) - that's the seek-sampling dependency, and it's genuinely needed here, not dead weight. And the pack still won't download llama.cpp itself; a llama-server build on PATH is a hard prerequisite for any of the Local AI nodes.