My Ollama Vision

MyOllamaVision is the reverse-engineering node in the ComfyUi-Ollama-YN pack: feed it an image, get a prompt back. It's the local, Ollama-powered cousin of a WD14 tagger or a BLIP captioner - except it's a proper vision LLM doing the describing, so it comes out as a sentence you can drop straight into your prompt, not a bucket of tags.

The most obvious use is the one the pack's README leads with: you have a picture (reference art, a screenshot, something you generated elsewhere) and you want to know what prompt would reproduce it. Wire the output into CLIPTextEncode and you've got a legit img2img or "describe then regenerate" loop. It's also handy for captioning batches of reference images before training a LoRA - though for that job you'll usually want something structured like booru tags, not prose.

How it works

The node takes your ComfyUI IMAGE tensor, converts each frame to a PNG in memory, base64-encodes it, and sends it to your local Ollama server via client.generate(images=...). The model you pick needs to be a vision model - llama, llava, and the other multimodal builds in the Ollama library qualify; pure text models will just complain about not receiving usable input. The query field is the instruction, defaulting to "describe the image".

The pack tracks vision models separately from text models via file/category.csv, so the dropdown distinguishes them with a "(vision)" suffix. First time you run with a new model it does an ollama pull and logs it to the CSV.

The inputs that matter

• images - a standard IMAGE input; connect any Load Image or anything that outputs an image tensor.
• query - what you want it to do with the picture. Default is "describe the image," but try asking it to focus on lighting, composition, or "write a detailed SD prompt for this." The quality of what you get back tracks how specific you are here.
• model / extra_model - pick a vision model from the dropdown, or type a new one into extra_model (remember the :latest suffix, and check the Ollama library that the model is actually multimodal).
• seed - vision calls get a seed too; bump it for variation if two runs give identical captions.
• keep_alive - 0 unloads the vision model from VRAM immediately after the call; 60m keeps it resident for repeat runs.

The single output, description, is a STRING - wire it into CLIPTextEncode's text, or into any node that takes a prompt. For an SD prompt you'll often want to chain this into My Ollama Generate Advance: caption the image, then have the text model turn the caption into a more SD-shaped prompt.

Installing it

Same pack, same drill. ComfyUI Manager (search "ComfyUi-Ollama-YN"), or:

cd ComfyUI/custom_nodes
git clone https://github.com/wujm424606/ComfyUi-Ollama-YN.git

Requirements are just ollama, pandas, and aiohttp. The real prerequisite is Ollama itself - installed, running, and reachable at the url (default http://127.0.0.1:11434). No API key anywhere in this stack.

Where people get burned

The usual suspects apply. Newly pulled models won't show in the dropdown until you re-add the node or restart ComfyUI (right-click "Reload" is the README's documented no-no). If the dropdown is empty, Ollama may not be running - the list is built live from ollama list. And if the model you picked is text-only, you'll get a garbled or empty response instead of a caption; check the model card on the Ollama library before blaming the node. If you want to see exactly what's being sent, flip debug to enable and the request details print to the ComfyUI console.

One honest caveat: like most local vision LLMs, the captions are good, not great. Treat the output as a starting point you'll edit, not a prompt you paste blindly.