Gemini 3 多模态分析

Here's the node that turns your ComfyUI graph into a vision-language model. Gemini3MultiModalAnalysis takes up to four images, a video, and an audio clip, jams them all into one API call alongside your prompt, and returns a single analyzed response. It's the pack's dedicated captioner/analyzer - the thing you reach for when you want Gemini to actually look at what your pipeline produced, not just read text about it.

This is the same family as JoyCaption and Florence-2, but on a frontier API instead of a local model (llm-in-comfyui.md has the full captioner landscape). The payoff is real quality and true cross-modal reasoning - compare two images, transcribe speech, describe a video's scene changes, all in one answer. The cost is the usual API trade: media leaves your machine, it's metered, and Google's filter applies to what you send and get back.

How it works

Images arrive as IMAGE tensors straight from any ComfyUI loader, get converted to base64 PNGs, and go in as inline parts. Video and audio are the fiddly part: they're VIDEO/AUDIO objects, and the node has to reverse-engineer a file path from them - it pokes at attributes like file_path, filename, path, handles VHS-style dicts, and falls back to treating the input as a string path. If it can't find a real file on disk, it errors out.

media_resolution behaves like the rest of the pack: set it to anything but "Auto" and the request routes to Google's v1alpha endpoint, which enables per-media token budgeting (1120 tokens for a high-res image, 70–280 per video frame). The default prompt in the node already asks for image description, multi-image comparison, video scene analysis, audio transcription, and a cross-media summary - you'll usually replace it with your own instruction, but it's a good template for what the node is for.

Inputs and outputs that matter

Required: prompt, the four standard auth fields, thinking_level, media_resolution, max_output_tokens, and temperature. Optional: image_1 through image_4, video, audio, and system_instruction.

The critical rule: at least one media input must be connected. This node returns an error if you run it text-only - it's not the text-generation node, it's the analysis node, and it knows its lane.

Outputs: response (the analysis), thought_signature (feed it forward to a later Gemini 3 call to keep context), and usage_info (the full token-count JSON from Google, including how many tokens the media consumed - worth a look the first time to see what a video costs you).

Install

No models, all cloud. ComfyUI Manager (search "ComfyUI-Gemini-3") or:

cd ComfyUI/custom_nodes
git clone https://github.com/xuchenxu168/ComfyUI-Gemini-3
cd ComfyUI-Gemini-3
pip install -r requirements.txt

You need a Google AI Studio key - api_key field, config.json, or GEMINI_API_KEY.

Common issues

• "Video path not found" - the node depends on your video loader exposing a real filesystem path. If you feed it a VIDEO object from an exotic loader, path extraction can fail; check that the node can see the file.
• Audio fails to encode - audio tensors go through scipy.io.wavfile, and scipy is missing from the pack's requirements.txt. pip install scipy if you hit it.
• Cost surprise - video analysis at high resolution is 280 tokens per frame, and a long clip will dominate your bill. Drop media_resolution to low for general description; reserve high for when you need the text read.
• The filter applies - Google's safety settings gate what this node will analyze and say. There's a dedicated safety-settings node in the pack if you want to loosen the thresholds, but you can't go beyond what Google allows at the model level.