🖼️ ImageToPrompt Ollama

Feed it an image - either a normal ComfyUI IMAGE tensor or a whole folder of them - and it hands back a descriptive prompt written by a local vision model running through Ollama. No API key, no cloud, no per-image billing. This is the pack's image-to-prompt node, built for the job the community actually does with VLMs: captioning LoRA training sets, and seeding prompts for img2img or image-to-video from an existing image.

The local thing matters more than it sounds like it does. Vision captioning is the one job where people consistently don't want a filtered API in the loop, and a small Qwen or Gemma on your own card is free per call and doesn't scrub content. That's the whole pitch of this node - it's the "general-purpose VLM via Ollama" end of the captioner spectrum, heavier and more flexible than a dedicated tagger.

How it works

You have two input paths, and you pick one. Connect an image tensor to image_1 and it processes whatever is upstream - a Load Image node, or the output of a previous workflow stage. Or flip use_batch_folder on and give it a batch_folder_path; it scans the folder for .png/.jpg/.jpeg/.webp/.bmp, and processes them in sequence.

Under the hood it does the boring-but-important prep work so local models don't choke. Transparent PNGs get their alpha pasted onto a clean white background (vision models misread transparency), non-standard color channels get normalized, and images are smart-downscaled to a max 1280px side before base64 encoding - the author's stated reason being that huge base64 payloads over raw HTTP are how local memory leaks happen. The endpoint logic is tuned too: apiMode defaults to auto, which tries /api/generate first, because vision models in Ollama answer far more reliably through generate than through chat wrappers - chat mode is notorious for replying while silently ignoring the image. A system prompt tells the model to describe only what's visible, never invent, and return one clean prompt.

The rest is configured in the app via vision_settings_json - model, system prompt, instruction, temperature - and saved as presets on disk under vision_presets/. The pack ships three tuned for qwen3-vl:8b: a precise captioner, a composition-focused one, and a graphic-style one.

The outputs that matter

• prompt (STRING) - the caption. Wire it into your text encoder to drive img2img or an image-to-video prompt assistant, or into an enhancer.
• prompts_json (STRING) - all captions as JSON, handy when you batch-caption and want them in one blob.
• image_count (INT) - how many images got processed this run.
• status_json (STRING) - per-run status: model used, batch vs. connected mode, whether a fallback kicked in, any error. Read this first when the node returns nothing.

Installing it

Pack install is the same as the rest of the pack (ComfyUI Manager → Orion4D MetaPrompt, or clone into custom_nodes, then restart), plus the Ollama prerequisite:

ollama serve
ollama pull qwen3-vl:4b    # the README's suggested size
# the shipped presets assume qwen3-vl:8b - pull that if you want them as-is

The node's default batch folder is {COMFY}/custom_nodes/Orion4D_MetaPrompt/vision_inputs - drop files there or point batch_folder_path anywhere, using {COMFY}/{CUSTOM} tokens.

Where people get burned

Multi-subject attribution is where every VLM fails, this one included - two people in the frame and it'll mix up who's wearing what. Audit captions by hand for small sets. The Qwen3-VL instruct models also carry assistant-style refusals, so for explicit captioning you'll want an abliterated build or a dedicated captioner like JoyCaption. And the honest gotcha: local VLMs are slow and hungry for VRAM, and a batch folder scan re-runs every queue - keep an eye on how many images you point it at, or your queue becomes a nap.