Gemini Video Captioner

The node that turns video into a usable caption

Video is where ComfyUI workflows get long, and captions are where they get useful. If you've generated a clip with an AnimateDiff/Wan-style pipeline or loaded one from disk, you often need to know what's actually in it - for tagging, for a description, for deciding whether it's worth keeping. GeminiVideoCaptioner is the pack's answer: give it a video file path or an image sequence, and it hands back a detailed caption plus a representative frame.

The clever bit is that it works two ways - a full video file or a stack of frames - so it slots into both "I have a file" and "I have frames in my graph" workflows.

How it works

Feed it either a video_path (a file on disk) or an image input (a batch treated as video frames). If you give it a file, it converts it to WebM (via moviepy/ffmpeg) and keeps it under the API's 30MB limit, falling back to frame extraction if the conversion fails. The frames_per_second setting controls sampling - 1.0 (default) means roughly one frame per second of video, which is enough for most scene description. max_duration_minutes caps how much video is analyzed (the source enforces 2 minutes for Gemini 1.0-class models, up to 45 for 1.5+).

process_audio (default "false") additionally sends audio for models that support it. There's also use_structured_output plus an output_schema - flip that on and the caption comes back as structured JSON (description, objects, actions, mood, technical details) instead of prose, which plays beautifully with the pack's JSON utilities downstream.

Inputs you'll set

• video_path or image - one or the other. File for real videos; frames for in-graph sequences.
• prompt - the analysis instruction. Default: "Describe this video scene in detail..." - change it to ask for timestamps, technical analysis, or safety flags.
• frames_per_second - 0.1–10. More for action-heavy footage, less for slow scenes.
• max_duration_minutes - cap runtime for long videos.
• model - gemini-2.0-flash default; gemini-1.5-pro for deeper analysis. The pack's docs explicitly steer you away from gemini-2.5-flash-lite for video.
• process_audio, use_structured_output, output_schema - the extras worth exploring once basics work.

Outputs: caption (the text), sampled_frame (an IMAGE preview frame - nice for a visual in the graph), raw_json (for structured mode), and the standard api_request / api_response debug pair.

Install

cd ComfyUI/custom_nodes
git clone https://github.com/jqy-yo/comfyui-gemini-nodes
cd comfyui-gemini-nodes
pip install -r requirements.txt

Or via ComfyUI Manager ("ComfyUI Gemini Nodes"), restart. This node is the reason the pack pulls in moviepy and opencv-python - that's where the video conversion happens.

Where people get burned

Video is the pack's most failure-prone area, and the README's troubleshooting is blunt about it. The classic symptom is 500 INTERNAL on video calls: that's usually regional restriction or your key lacking video-model access - test with a tiny clip and a basic model (gemini-1.5-flash) first. Keep files under 30MB and short (<10 seconds for a smoke test); big inputs reliably 500. If video keeps failing, the documented fallback is frame extraction: pull frames yourself and run them through the image nodes instead. And don't send a 45-minute file expecting magic - that's the maximum cap, not a recommendation, and it will burn tokens and time.