Gemini Chat (Vertex AI)

The most quietly useful node in this pack, and probably the one you'll use most. Gemini Chat puts a full multimodal LLM in your graph: describe an image, get a prompt back. Ask for a caption, feed the result into a prompt box. Rewrite a weak prompt into something a checkpoint actually understands. It's the Swiss-army text node that turns ComfyUI from a generation tool into something that can think about what you're generating.

What it is

A text/generation node that calls Gemini through Vertex AI. It's a single-turn chat (no conversation memory - you give it a prompt, it gives you a response), but the inputs make it far more than a text box: it accepts an IMAGE and an AUDIO input alongside the prompt, so you can ask Gemini what's in a frame, or have it transcribe and summarize an audio clip, then hand the result to anything downstream.

How it works

The node assembles a generate_content request. Your image becomes a PNG part, your audio gets re-encoded to a WAV part, and everything rides to Vertex AI together with the prompt. Responses come back as a plain string. Two inputs that change the behavior in interesting ways:

• google_search - lets Gemini consult live web search before answering. Genuinely useful for "what's the current model called" style questions mid-workflow.
• url_context - lets it pull context from URLs you mention.
• thinking / thinking_budget - reasoning mode. This is where it gets fiddly: Pro models cannot have thinking turned off. If you pick gemini-2.5-pro, the node silently forces a thinking budget even when thinking is false, so don't expect Pro to behave like a snappy Flash.

The inputs that matter

The usual credential trio - project_id, location, service_account - and yes, the same pack-wide gotcha: service_account wants the contents of your JSON key file pasted in, not a path.

• prompt - the thing you're asking. Multi-line, so pasting a chunk of source or a long image description is fine.
• model - the dropdown runs the 2.0/2.5 line plus whatever previews are current (the source code even lists newer 3.x names). gemini-2.5-flash is the sensible default; reach for Pro only when the answer needs real reasoning.
• system_instruction - optional. Set this to "you are a prompt engineer" and the node becomes a prompt-rewriter; set it to "describe this image in detail" and it becomes a captioner.
• temperature (default 0.2) and top_p - keep the temperature low for captioning and prompt work; you want consistency, not creativity.

Outputs

response - a single STRING. It wires straight into any prompt input in ComfyUI, or into text-processing nodes. The classic pattern: image → Gemini Chat ("describe this") → response → your text-to-image node's prompt.

Installing

cd ComfyUI/custom_nodes/
git clone https://github.com/Aryan185/ComfyUI-VertexAPI.git
cd ComfyUI-VertexAPI
pip install -r requirements.txt

Restart ComfyUI, or grab ComfyUI-VertexAPI through ComfyUI Manager.

Where people get burned

Token costs are real but small - this is cheap unless you loop it thousands of times. The bigger gotcha is the Pro thinking quirk above: if your outputs look slower and pricier than expected, check whether you've got a Pro model selected. And because the service account does everything, one mis-pasted JSON means every node in this pack fails the same way at once - fix it once in the field and it propagates to all of them. Set up the credentials properly one time and Gemini Chat becomes the node you reach for constantly.