Ollama VL → Prompt

Want a caption for a reference image without paying a cloud API? Ollama VL → Prompt is the local answer. It takes up to three images (or just a text idea), sends them to any vision-language model you already have in Ollama, and hands back a prompt string you can feed straight into a CLIP text encoder. No key, no account, no data leaving your machine - Ollama does the vision work locally, on GPU or plain CPU.

This is exactly the kind of node that's become quietly important since the big models moved to LLM text encoders. Z-Image, Flux 2 Klein, and Anima want flowing natural-language descriptions, not comma-separated tags. Feeding them a captioned reference is a real workflow - and an image-to-prompt node is the fastest way to get there.

How it works

The mechanism is simple and easy to audit. The node converts each incoming image tensor to a base64-encoded PNG, then calls Ollama's chat API with the image(s) plus a system prompt picked from presets.json. The model's reply comes back as a string.

Two image modes, controlled by one toggle:

• combine_all_images = False (default): one Ollama call per image, returning one prompt per image in a list.
• combine_all_images = True: all images go into a single call as combined context, and newlines get collapsed to commas in the result.

No images connected at all? It falls back to a text-only chat - handy for pushing an existing prompt through a model with just a user_hint.

The inputs that matter

• model - auto-populated from ollama list when the node loads. All your Ollama models show up, including text-only ones, so pick a vision model (llava, moondream, qwen2-vl, llama3.2-vision). Grab one with ollama pull llava.
• preset - seven built-in system prompts: SDXL, Inpainting (Preserve Context), Anime / Illustration, Caption (Neutral), Qwen Image Edit 2511, NSFW, SFW. They're editable in presets.json, which is a nice touch.
• user_hint - optional extra instruction appended to the preset ("make it moody", "keep the palette").
• keep_alive - minutes the model stays warm in VRAM. 5 is fine; raise it if you're running batches.
• image1 / image2 / image3 - IMAGE tensors, up to three.

The one thing that trips people up

The output is always a list. generated_prompt comes out as a STRING list even when there's a single image, so you can't wire it straight into a text encoder. Run it through a list-to-string or "get first item" node first. Combine with the fact that a failed Ollama call isn't an exception - errors get baked into the output string as "Ollama error: ..." - and you can silently ship garbage to your sampler. Watch the console for the print statements.

Install

ComfyUI Manager: search comfyui_ollama_vl_prompt and install. Or by hand:

cd ComfyUI/custom_nodes
git clone https://github.com/JuanBerta/comfyui_ollama_vl_prompt
cd comfyui_ollama_vl_prompt
pip install -r requirements.txt

Then restart ComfyUI, make sure Ollama is running (ollama list should show models), and pull a vision model: ollama pull llava or ollama pull moondream. If the model dropdown reads "No model detected", Ollama isn't running or has no models - and since the list is read at node load, restart ComfyUI after pulling.

Two real-world caveats. First, requirements.txt pins versions (numpy 2.4.2, pydantic 2.12.x), and pip will happily shuffle your shared Python env to satisfy them - the classic custom-node dependency dance that breaks other nodes. Second, on the security front: after the ComfyUI_LLMVISION malware incident, any LLM-vision node deserves a skeptical look. This one is a genuine small pack - roughly one readable Python file that only talks to your local Ollama, no remote endpoints, no hidden wheels. That's the model of custom node worth trusting, but the rule stands: read before you install.

It's a niche tool with a light footprint - no heavy model downloads of its own, since the vision models live in Ollama. If you already run Ollama and you've ever hand-typed a caption into a text encoder, this saves you the typing.