Gemma 4 Reference Prompt

If you do Wan or VACE image-to-video, you know the drill: you've got a reference image, and you need a prompt that actually describes it the way the model cares about - materials, lighting, camera, environment, motion-ready details - not the way a human would. That's exactly what Gemma4ReferencePrompt does. Feed it an image, it runs a small multimodal model on your own GPU, and hands you a prompt string you can drop straight into your I2V workflow.

The name is a lie in the best way: it doesn't call any API and needs no key. The "Gemma" is Google's instruction-tuned model, loaded locally through HuggingFace Transformers. You get a vision-language model that's actually looking at your reference image instead of you typing from a screenshot.

How it works

Under the hood it's straightforward: the node feeds your image plus an instruction into AutoProcessor + AutoModelForMultimodalLM from google/gemma-4-E2B-it (the default model_id, an efficient ~2B variant; E4B is the bigger, slower sibling), decodes the generated tokens, and outputs plain text. The model is cached in RAM per model_id/dtype combo, so repeat runs don't reload it. Set temperature to 0 and it goes fully greedy - same image, same prompt, every time.

The inputs that matter

Most of the defaults are sensible for Wan/VACE work, and you'll rarely touch more than a few:

• image - any IMAGE tensor, straight from a Load Image node.
• instruction - multiline text. The default is already the Wan/VACE recipe (subject, materials, lighting, camera, environment, motion). This is the one you'll edit when you move to a different model family or want a different voice.
• model_id - google/gemma-4-E2B-it (default) or google/gemma-4-E4B-it. E4B is heavier; only bother if the 2B output reads too shallow.
• temperature - 0.2 default. Low is good here; you want stable captions, not poetry. 0 is greedy.
• unload_after_generate - default true, and leave it that way. It clears the model from VRAM after each run, which matters in a video pipeline that's already hungry for memory.
• max_new_tokens (384 default) and dtype (auto) are the "leave them alone" ones.

It returns a single prompt STRING, which wires straight into the CLIP Text Encode positive input of your Wan or VACE workflow.

Installing it

ComfyUI Manager is the easy path: search for ComfyUI Gemma4 Reference Prompt, install, install requirements when Manager asks, then fully restart ComfyUI. Or:

cd ComfyUI/custom_nodes
git clone https://github.com/MaikiOS/ComfyUI-Gemma4-ReferencePrompt
python -m pip install -r ComfyUI-Gemma4-ReferencePrompt/requirements.txt

The one heavy dependency is the model itself. It's not in ComfyUI/models - Transformers downloads it to your HuggingFace cache (typically ~/.cache/huggingface) on first run, so expect a wait and a few GB of disk the first time you hit run. If HuggingFace asks for a token, huggingface-cli login sorts it.

Where people get burned

The classic failure is Unrecognized processing class, which means your Transformers is too old for the Gemma 4 classes. Fix it:

python -m pip install --upgrade "transformers>=5.5.0" accelerate

The >=5.5.0 floor is strict, and Transformers is one of the packages most likely to collide with other custom nodes in ComfyUI's shared environment - if upgrading fixes this node and breaks another, that's the usual dependency-hell tradeoff, not a bug in this pack. Beyond that, the first run just takes a while (model download + load), and on a small card the load itself can churn VRAM until unload_after_generate clears it. None of it is exotic, but all of it is more likely to bite you than the node itself.