Ollama Audio Chat

Ollama Audio Chat lets you drop an audio clip into a ComfyUI graph and have a local model actually listen to it - "what's that sound in the background?", "summarize this voice memo", "transcribe the dialogue and tell me who's speaking." It's the audio sibling of the pack's vision nodes, and the only reason it exists is a hack: Ollama still has no native audio API, so the node shoves your sound through the image channel and Ollama quietly routes it to the model's audio encoder. It works, it's free per call, and there's no cloud and no key.

You reach for this when you already have an AUDIO output in the graph - from ComfyUI's LoadAudio, a music-generation workflow, whatever - and you want a model to make sense of it without exporting a file and dragging it into some other app. It's the same "small local LLM as a tool node" pattern as the pack's text and vision nodes, just pointed at your ears instead of your eyes.

How it works

Behind the scenes it's an ollama.chat() call. Your audio gets converted to 16kHz mono 16-bit WAV with a full RIFF header - that header is the load-bearing part; Ollama sniffs the RIFF/WAVE magic bytes in the images[] field and sends the data to the model's audio encoder, and raw PCM without the header fails silently. The node handles resampling and the mono mix itself, so you don't think about any of that.

Two things it does automatically that will save you a support ticket:

• Audio goes first in the images list, before any image frames, because that's where Ollama expects it.
• When audio is connected, num_ctx is capped at 8192 - audio embeddings blow past that limit and overflow memory, so it enforces the cap for you.

It also passes through the think flag for chain-of-thought on models that support it (most won't), and returns a meta output so you can chain connectivity and options into the next Ollama node.

The inputs that matter

• prompt - the text you send alongside the audio. Defaults to "Describe what you hear in this audio."
• audio (optional) - connect any AUDIO output. Leave it empty and the node just runs as a text-only chat, which is a perfectly fine way to use it.
• connectivity - an Ollama Connectivity node carrying your server URL, model, and keep-alive. This or meta is required; connect neither and the node raises an error the moment you run it.
• system - the system prompt, "You are a helpful assistant." by default.

Outputs: result (the model's answer as a plain STRING), thinking (scratch text when think is on), and meta for chaining.

The model is the whole game

The node sends audio, but the model has to actually accept it. The names to pull into Ollama are gemma3n and qwen2-audio - ollama pull gemma3n. A plain text model like llama3.2 will cheerfully tell you it can't hear anything, because it can't.

Install and run

This ships in the ComfyUI-OllamaOmni pack. Easiest via ComfyUI Manager - search "OllamaOmni" - or manually:

cd ComfyUI/custom_nodes
git clone https://github.com/ckinpdx/ComfyUI-OllamaOmni
pip install -r ComfyUI-OllamaOmni/requirements.txt

Restart ComfyUI, and make sure Ollama is running and reachable from the ComfyUI host. That requirements file is a single line - ollama - so there's no heavy dependency surprise.

Where people get burned

• Wrong model. If the output is a polite apology instead of an answer, you're pointed at a text-only model.
• No connectivity/meta. This node is the picky one: it won't guess a server, so wire up Ollama Connectivity first.
• Long audio + memory. The enforced num_ctx cap keeps you under the audio limit, but very long clips still cost context. Keep clips short and focused.

As with any LLM node, you're running arbitrary Python from a GitHub repo on import - use the widely-installed, open pack, and glance at what it does before you add it to your default setup.