Vision LLM

What this node actually is

One node that sends your image plus a text prompt to whatever OpenAI-compatible vision endpoint you point it at, and hands back the model's text. No weights to download, nothing runs on your GPU. This is the LLM-as-a-tool pattern, not a diffusion node.

So it works upstream of your sampler and will never make an image better directly. What it's for: captioning a pile of images into a LoRA dataset, writing a description of a reference photo to seed img2img, describing a clip's first frame for image-to-video, or acting as a checker in a loop. It does plain text work too - image is optional, so it's a general chat node with the same profile management.

Before you use it to caption a training set, weigh the local option: an 8B abliterated model is enough for captioning, and it's uncensored, offline, and free per call. JoyCaption exists precisely because the alternative was an API that was "expensive and heavily censored." Take the API path for frontier description quality when local or NSFW doesn't matter. Then accept the trade: it costs per call, the vendor's filter applies and you can't abliterate a closed model, and your image leaves your machine. That's the mechanism, not a bug.

How it works

Each profile's base URL gets /chat/completions appended for generation, and /models for the connection tester. No openai pip package, just aiohttp, which ComfyUI already ships. So the base URL must include the vendor's version prefix (https://host/v1) - the code only appends the path.

Your image goes out as a base64 data URL - normally a lossless PNG, which is the right default. Except it silently re-encodes when things get big: past 2048px on either edge, 6 megapixels, or an 8 MiB PNG, it Lanczos-resizes down and switches to quality-90 JPEG. It prints a log line when it does (in Chinese - 大图自动优化 - as are several of its error strings, which is normal for this pack, not a broken install).

Since a ComfyUI IMAGE is batched, the node puts every frame into a single user message as separate image_url parts, so one call can compare several frames. system_prompt becomes a real system message, and pre_prompt is prepended to text_prompt with a newline: a cheap way to keep a fixed instruction in front of the part you edit.

The return path is forgiving: it accepts OpenAI's choices[0].message.content, DeepSeek-style reasoning_content, Gemini's native candidates[].content.parts (with thought parts routed to reasoning), or a bare output_text. Safety blocks surface as a real error with the reason, not an empty string.

The inputs worth setting

text_prompt is the ask. model is a plain text box, not a dropdown - leave it blank and it falls back to the first model in your profile's list. profile picks which saved API config to use (auto = whichever is active).

temperature and max_tokens each have a paired enable toggle, and both default to off - so those values are not sent at all and the server's default applies. If the temperature slider looks ignored, that's why.

strip_reasoning is on by default: thinking models that inline <think>...</think> in their answer get that text moved into the second output instead of your prompt. Change the reasoning_tag_open/reasoning_tag_close pair if your model uses different markers. image_detail passes OpenAI's detail hint; gateways that don't know the field ignore it. sleep waits N seconds after the call - for a same-machine LM Studio or Ollama setup, giving the local server room to breathe.

Two outputs: text (wire it into a CLIPTextEncode, a save-text node, or another LLM node) and reasoning (usually you just want to peek at it).

Install

ComfyUI Manager → search Vision LLM, install, restart. Or:

cd ComfyUI/custom_nodes
git clone https://github.com/yurishk/ComfyUI-VisionLLM
# restart ComfyUI

No pip installs, no model files - unusual and welcome. Then click ⚙ API Settings, add a profile (base URL including /v1, API key, candidate models), mark it active, test the connection. Credentials live in custom_nodes/ComfyUI-VisionLLM/config/api_settings.json, git-ignored; the UI only ever sees a masked key.

That's the whole reason this node exists: your key isn't a widget, so it isn't serialized into the workflow JSON, and sharing the workflow leaks nothing. Most API packs put the key in a node field or a config.ini, and a workflow with your key baked in is an easy way to hand a stranger your billing. The settings routes return masked keys only, too - which matters, since ComfyUI's API is unauthenticated by default.

Where people get burned

It ran once and then "stopped working." IS_CHANGED hashes every input, so identical inputs mean the node is cached and never re-calls the API. Nudge the seed (its control_after_generate widget exists for this) to force a fresh call.

404 or 401. Nine times out of ten the base URL is missing /v1; otherwise the key is wrong. Errors are sanitized, so you'll see Bearer *** rather than your key.

A big image quietly isn't lossless. For pixel-level questions ("is this line perfectly straight?") the auto-downscale to JPEG throws away the detail you're asking about. Crop first.

A 300-second hard timeout is baked into the client with no widget for it, so long generations on a slow local model die on it. Enable max_tokens and keep it modest behind a gateway - the pack's own error message notes that a passing model-list test proves nothing about long generations.

Cancel is honest; the provider may not be. Interrupting aborts the local HTTP request instead of waiting out the timeout, but a third-party endpoint can still finish and bill work it already accepted.

Trust, briefly. The exact string "VisionLLM" turns up in three community threads ever, none since early 2025. Nobody is reviewing this pack for you - and the one time an "LLM vision" node went bad here, it was credential-stealing malware that ended in a federal prosecution. The mitigating fact: a few hundred lines of readable Python, zero dependencies, zero bundled downloads. Read it before you run it.