Video Captioner (Advanced)

Video Captioner (Advanced) is the full-control version of the video captioning workflow, and it's where the pack's smartest mechanism lives: it decides how the VLM sees your video based on what the backend actually supports, instead of assuming. The simple Video Captioner sets sensible defaults; this node exposes every knob, and it's worth understanding even if you never touch it, because the info output tells you what path your video actually took.

The video_mode input is the headline feature:

• auto - the default. The node checks llama-server's /props endpoint. If the backend explicitly advertises typed native video (input_video), it sends the video natively; otherwise it falls back to sampled frames. Missing or incomplete capability metadata is treated as unknown, and unknown defaults to sampling - conservative by design.
• native_video - force native video. Errors if the backend reports no support; if support is unknown, it attempts it anyway and can fail.
• sampled_frames - force the sampling path: the clip is decoded into timestamped JPEG frames and sent as an image sequence. Works with any vision-capable model.

Because native video depends on both the model/projector and your llama.cpp build, auto mode quietly saves you from a whole class of "why is my video request rejected" confusion - and when it does fall back, the info output reports native_fallback=<reason>.

When sampling, two inputs control the extraction. sampling_mode is uniform (spread sample_frames evenly across the clip) or fixed_fps (grab at sample_fps, up to max_frames). Defaults: 12 samples at 2 fps, capped at 24 frames. The sample settings matter more than they look: too few frames and the model can't see the action, too many and you blow the token budget. For file-backed clips the extraction uses PyAV seek sampling, so memory tracks the sampled frames, not the total clip length - a real virtue on long source footage.

The rest is the captioner panel you'd expect: instruction, caption_style (same six options as the image captioner plus video-aware wording - no invented audio or dialogue), max_tokens, temperature, seed, and optional reasoning_effort, system_prompt_override, caption_prefix, banned_phrases, image_max_edge, jpeg_quality. Note the two outputs: caption and info. info is genuinely useful - it reports mode, fps, duration, frame count, how many frames were sampled, their timestamps, and the fallback reason if any. When a caption comes out wrong, that string is where you start diagnosing.

Same prerequisites as every captioner: the config from Local AI Model (Advanced) must have a matching mmproj projector, or the node errors before any video processing. And the "no audio/dialogue inference" rule holds here too - these captions describe what's visible, period.

Install is the pack standard: ComfyUI Manager, search WepeNerd, or git clone https://github.com/WepeNerd/ComfyUI-WepeNerd.git into custom_nodes, pip install -r requirements.txt (the av dependency is real and used), restart. llama-server on PATH is the usual hard requirement.