Ollama Vision Captioner (Nukun)

Point this node at any image in your graph and it runs a local Ollama vision model over it, returning a caption, a tag list, a ready-to-refine "seed" string, and a detail-pass prompt for HiResFix. The headline use case is the one the README spells out: IMAGE -> Ollama Vision Captioner -> Ollama Prompt Refiner. You generate a rough image, the vision model describes what it actually produced, and the refiner turns that into a prompt for the next pass. That's the img2img-to-prompt loop, and it's far more useful than it sounds - captioning a real output catches details you'd never think to prompt for.

It's also a legitimately safe way to do this job, which matters because captioning nodes have been weaponized before in this ecosystem. Backend v1 uses only Ollama's /api/generate image support - your image is sent to your own Ollama server at 127.0.0.1, nothing leaves the machine, and no Transformers or llama-cpp-python get dragged in. No API key, no mysterious network calls. This is the right shape for a local captioner.

How it works

Your ComfyUI image tensor gets converted to RGB JPEG, downscaled to resize_long_edge (1024 by default, preserving aspect ratio), base64-encoded, and sent in the Ollama images field. The default model is user-v4/joycaption-beta - JoyCaption, the community's default VLM captioner, which the KB flags as the best-of-breed natural-language captioner for this exact job. You can also run JoyCaption Alpha Two (selected or typed as hf.co/Jobaar/Llama-JoyCaption-Alpha-Two-GGUF:F16) or any vision model you've pulled.

caption_mode decides the output shape:

• natural_caption - two to four readable sentences plus a tag list.
• danbooru_tags - 24–48 comma-separated booru-style tags, for Illustrious/Pony-style bases.
• pony_source - 35–65 comma-free factual words for Pony v6 / Illustrious workflows.
• refiner_seed (default) - the richest comma-free 40–80 word seed for the Prompt Refiner, with model-control tags like score_9 and style_cluster_* stripped out.

The five outputs map to the refiner: text_seed goes to its word_salad (or use it as a richer style_anchor), caption is the readable description, tags is the booru list, hiresfix_text is a detail-pass prompt with material cues ("fluffy fur", "leather grain", "fine hair strands") added when the caption matches, and report tells you what happened. Note v1's batch behavior is intentionally simple: only image[0] gets captioned, and the report mentions it when your batch had more.

Installing it

Pack install, then Ollama plus a vision model:

cd ComfyUI/custom_nodes
git clone https://github.com/OnekoSL/Nukun_ComfyUI_Nodes.git

Restart ComfyUI. On the Ollama side: ollama pull user-v4/joycaption-beta, or point ollama_model at any vision-capable model. If a model doesn't show in the dropdown yet, the node keeps your typed value, so you can paste a name while the pull finishes. unload_after_run is on by default - the model unloads after captioning so the next diffusion pass gets the VRAM back. Leave it on.

Common issues

The most common failure is using a text-only model - you'll get an Ollama error or a caption that ignores the image entirely. Check your model is actually vision-capable. Long captions can exceed small context windows: context_length defaults to 4096, which is fine for JoyCaption, but raise it if you're running a bigger VLM that truncates. And the multi-subject weakness is real - VLMs mix up who's wearing what when there are several characters in frame, so treat the caption as a starting point for complex scenes, not ground truth. Malformed JSON gets one text-only repair retry, then a local fallback built from the raw response so your workflow still gets usable text.