Gemma 4 Image Understanding

You've got an image in ComfyUI and you want a sentence about it - a caption for a LoRA set, a description to feed an image-to-video prompt, a sanity check that the character actually matches. This node hands the image to Google's Gemma 4 12B multimodal model running locally, and returns what it sees as a string you can wire anywhere. No API key, no cloud, no subscription. The name is the whole pitch: it's Gemma 4, it understands images, and it lives on your card.

This is the middle of the VLM-in-the-graph pattern, the same job JoyCaption and Florence-2 do - with a bigger, more literate model behind it. The tradeoff is obvious once you've read the specs: Gemma 4 12B is not a 0.77B Florence. It's slower, hungrier, and far more descriptive. Reach for it when you want rich natural language rather than fast tags.

How it works

The node loads gemma-4-12b-it from ComfyUI/models/LLM/gemma-4-12b-it/ using the official ModelScope transformers path (AutoModelForMultimodalLM + AutoProcessor) - not llama.cpp, not Ollama. That detail matters more than it sounds. Gemma 4's new unified architecture broke a bunch of ComfyUI loaders at launch; this pack sidesteps the whole mess by using the official implementation, and you pay for that safety in VRAM (more below).

One mechanism worth knowing: the node decides what you fed it by the batch size. A single image goes in as an image. An image batch of more than one frame gets treated as sequential video frames instead. So this node is really "image understanding, plus a free multi-frame mode" - feed it a batch and it describes the sequence.

The inputs that matter

Most of the sliders are standard sampling controls you can leave alone. The ones you'll actually touch:

• image - the IMAGE tensor. One image, or a batch of frames.
• text - your question or caption instruction ("Describe this image in detail." by default). This is the user message; the model answers it.
• prompt - an optional system prompt, the author's way of steering Gemma's tone before the question.
• vision_token_budget - how many tokens the vision encoder spends per image. 280 default; step it up to 560/1120 for detail-hungry work, drop to 70/140 for speed. Higher costs VRAM and time.
• enable_thinking - when on, Gemma reasons first and the node wraps that in <think>...</think> before the actual answer. Slower, and you'll usually strip it before the output feeds a prompt.
• keep_model_loaded - default true keeps the ~24GB model resident between runs. Turn it off and it unloads after each call, freeing the card for the sampler.

The output is a single RESPONSE string. Wire it into any string slot - a prompt builder, a save-text node, a caption file writer.

Installing it

Grab the pack through ComfyUI Manager (search "Gemma 4 - Multimodal AI"), or:

cd ComfyUI/custom_nodes
git clone https://github.com/mailzwj/ComfyUI-Gemma4
cd ComfyUI-Gemma4
pip install -r requirements.txt   # numpy, torch, Pillow, modelscope

Restart ComfyUI. Then the part that trips people: the model does not auto-download. Go to ModelScope, grab the repo, and drop it into ComfyUI/models/LLM/gemma-4-12b-it/ with the model.safetensors, config.json, tokenizer.json, and friends. First run errors with a load failure until that folder is populated.

Where people get burned

• VRAM. Full-precision 12B is roughly 24GB in bf16. On a 16GB card this does not coexist with a diffusion model - this is why the community's default for Gemma 4 is a 4-bit Ollama/GGUF build at ~6.75GB instead. If the model and the sampler are fighting for the card, turn off keep_model_loaded and run your LLM pass first.
• ModelScope as the source. The README points at ModelScope, not HuggingFace. If you only know the HF URL for Gemma, you'll be confused about where the files are supposed to come from.
• Slow first call. Loading a 12B model takes a while; every run after that is instant only while keep_model_loaded is on.

It's a young, small pack from an author who's basically unknown in the community, and it's executing arbitrary Python on import like every node does. Read before you trust, same as always. But if you want a genuinely local Gemma 4 that can see, this is one of the few paths that actually works end to end.