Gemma 4 Video Understanding

You generated a video, you want to know what it actually shows - or you want a description to loop back into an image-to-video prompt. This node feeds a sequence of frames to Gemma 4 12B and gets back a plain-language read of the clip. It's the video sibling in the ComfyUI-Gemma4 pack, and the trick it relies on is that ComfyUI already handles the video part: you give it frames, not a file.

That framing decides everything. The node doesn't decode a video, doesn't need ffmpeg, and doesn't have its own timeline. It takes an IMAGE batch - the same thing every video workflow already produces when it samples frames - and treats it as a filmstrip. If you've got frames coming out of a video loader or a frame-extraction node, you're already halfway to using this.

How it works

The model (gemma-4-12b-it from ModelScope, via the official transformers loader - the path that dodges the llama.cpp/GGUF breakage that plagued other Gemma 4 integrations at launch) receives your frame batch as a single "video" message. The interesting part is frame_count: default 16, max 128. If your incoming batch is bigger than that, the node subsamples evenly - np.linspace picks evenly-spaced indices, so you get a fair skim of the whole clip, not just the first 16 frames. It also tells the processor to use exactly the frame count you gave it, which keeps the token cost predictable.

That's the node's whole discipline: cap the frames, budget the vision tokens, describe. A 16-frame sample at the default vision_token_budget of 70 per frame is a lightweight ask compared to a full image captioning pass.

The inputs that matter

• video_frames - the IMAGE batch. Frames in order. Upstream you want a node that samples a video to frames.
• frame_count - 16 default. More frames = better coverage of a long clip, more tokens and VRAM. For a short clip, 16 is plenty.
• text - your question about the video ("Describe this video." by default, but "What is the camera doing in this shot?" works far better for prompting).
• prompt - optional system prompt for tone and format.
• vision_token_budget - vision tokens per frame. 70 default here (lower than the image node, because you have many frames). Raise it only if the descriptions feel shallow.
• enable_thinking - Gemma's reasoning pass, wrapped in <think>...</think>. Useful for genuinely analytic questions about the clip; skip it for a plain description.
• keep_model_loaded - default true. Flip off to release ~24GB of VRAM after the call if the sampler needs the card.

Output is one RESPONSE string - a caption, a shot list, whatever you asked for.

Installing it

ComfyUI Manager (search "Gemma 4 - Multimodal AI"), or:

cd ComfyUI/custom_nodes
git clone https://github.com/mailzwj/ComfyUI-Gemma4
cd ComfyUI-Gemma4
pip install -r requirements.txt   # numpy, torch, Pillow, modelscope

Restart, then manually download the model from ModelScope into ComfyUI/models/LLM/gemma-4-12b-it/. No auto-download - an empty folder just fails on load.

Where people get burned

The heavy price is the same as the pack's: full-precision 12B is ~24GB in bf16, which is why most people run Gemma 4 as a 4-bit Ollama build instead; a 16GB card will feel this node squeezing the sampler. Second, remember the frames are on you - plug in a video loader or frame sampler upstream, or there's nothing to describe. And frame_count caps at 128: a 200-frame clip gets subsampled down, which is exactly what you want, but don't expect it to count the frames it didn't look at. It's a small, new, low-profile pack; skim the code before you trust it with your workflow, same as any fresh node.