API Gemini ImgOrAudioOrVideo2Text

API Gemini ImgOrAudioOrVideo2Text is the only node in this pack that takes all three media types at once: an image, an audio clip, or a video (as frames) - pick one or stack several - and returns text about it. If the pack's other API nodes each cover one modality, this is the one that covers the lot, through Gemini.

The use cases fall out of the name. Reverse-prompt a still, get a transcript or music description from audio, or describe motion from a video frame batch. Because it's Gemini you're also getting the strongest instruction-following of the pack's understanding nodes - good for "describe the lighting, then extract an SDXL prompt" in one call. All of it costs a per-call API meter and sends your media to Google, which is the standing tradeoff of every API node (external-api-nodes.md again), but for occasional multimodal understanding it's hard to beat one key and zero VRAM.

How it works

The node builds a Gemini request where your media are inlined: an image tensor becomes an inline PNG, an AUDIO input is converted to MP3 bytes (via pydub, which needs ffmpeg on your PATH), and a video is a batch of IMAGE frames, each sent as an inline PNG. Note the precedence: if a video batch is present it's used instead of a single image, and audio is appended separately - so image+audio in one call works, but image+video doesn't. The model dropdown carries seven Gemini choices (default gemini-2.5-pro-preview-05-06) and the seed is genuinely sent. Key: GOOGLE_API_KEY or the field.

The inputs that matter

• image (IMAGE, optional) - one frame/still.
• audio (AUDIO, optional) - a clip; converted to MP3 for the API.
• video (IMAGE, optional) - a batch of frames; this is how ComfyUI video flows through, and it replaces the single image when present.
• prompt (multiline) - the instruction; this node follows it well, so be specific.
• model - gemini-2.5-pro-preview-05-06 default; the list includes flash and 1.5 builds for cheaper runs.
• temperature / top_p / top_k / max_output_tokens / seed - sampling controls; the seed is actually honored here.
• proxy - the shared Gemini gotcha, below.

Output: a single text STRING.

Installing it

Ships in ComfyUI_Prompt-All-In-One (billwuhao). ComfyUI Manager → search Prompt-All-In-One, or git clone https://github.com/billwuhao/ComfyUI_Prompt-All-In-One.git into ComfyUI/custom_nodes, pip install -r requirements.txt, restart. It's under 🎤MW/MW-Prompt-All-In-One.

Where people get burned

Three things. First, the proxy field: same always-fires bug as the other Gemini nodes - it sets http_proxy/https_proxy on every run, and its default value http://127.0.0.1:None isn't a real proxy, so unexplained proxy errors trace back here. Second, ffmpeg: audio input converts through pydub, and a missing ffmpeg binary breaks exactly the audio path - check ffmpeg exists before assuming your audio clip is the problem. Third, frame count: video is sent as one inline image per frame, so a long clip makes a huge request - sample your frames down first. Audio and video both leave your machine, so keep the privacy and moderation reality of the API route in mind.