Image With Prompt

This node is a tiny bridge between your ComfyUI graph and OpenAI's vision model. You feed it an image and a text instruction, it phones home to GPT-4, and it hands you back a text string. No VRAM touched, no model files downloaded - just an API key and some patience while a server far away does the thinking.

Why would you want that in a graph? The classic use is captioning: drop a rendered image in, get a natural-language description back that you can paste into a training dataset or reuse as a prompt. The default prompt field is literally a captioning instruction, written in the style of a LoRA trainer - "the most important aspects of the image should be described first," with optional (word:weight) emphasis, a format the local tagger world borrowed from A1111's prompt weighting. It'll do a lot more than captions, though. Describe a character for you, write a prompt in your style, analyze the composition, turn a rough sketch description into a gen-ready sentence. It's just a chat-completions call wearing a node costume.

How it works

The mechanism is dead simple and fully in the open. The node converts your IMAGE tensor to a PIL image, base64-encodes it as JPEG, then calls chat.completions.create with a user message containing your text plus that image as a data:image/jpeg URL. Whatever the model says in choices[0].message.content is what you get out. The whole pack is one node, one class, and a couple of small helpers - there's no batching, no caching, no queue.

Two things worth knowing before you wire it up. The model name is hardcoded to gpt-4-vision-preview in the source, which was OpenAI's vision model from late 2023. That name is long since superseded by GPT-4o (which does vision natively), and if your key gets a "model not found" error, that's why - you'll be editing nodes.py to swap in gpt-4o. And the OpenAI client is created in the node's __init__, which means it reads your API key once when the node loads. Set the key before launching ComfyUI, or restart after changing it.

The inputs that matter

There are only three, so this stays honest:

• Image (IMAGE) - wire this from Load Image or anything that produces a tensor. This is what the model looks at.
• prompt (STRING, multiline) - the instruction. This is the entire personality of the node; change it and the node does something different.
• max_tokens (INT, 1–2048, default 77) - how many tokens the model may spend on the reply.

That default of 77 is a cute CLIP tribute and a real trap: it truncates long replies, and a decent caption will blow past it. Bump it to 400–800 if you actually want good output.

Output: a single STRING. Wire it into a text display or a Save Text node, or pipe it onward into a prompt builder. There's no image output - this node reads, it doesn't draw.

Install

ComfyUI Manager → search KepOpenAI, or:

cd ComfyUI/custom_nodes
git clone https://github.com/M1kep/ComfyUI-KepOpenAI

Restart ComfyUI. The only real dependency is the openai Python package (>=1.1), which the manager handles; manually, pip install -r requirements.txt in that folder. Then set the key, e.g. on Linux/macOS before launching:

export OPEN_AI_API_KEY=sk-...

The honest downsides

There's no widget for the key - it's env var or nothing, which trips up more people than it should. It's also paid and moderately censored: the vision API will refuse whole classes of images (including most NSFW) that your local stack wouldn't blink at, and a big captioning run costs real money per image. If your goal is bulk captioning for training, the community has moved on to free local models - JoyCaption and Florence 2 are the ones that keep coming up - and the local results are often better. Where this node still earns its place is the occasional "describe this one image" task where you'd rather not download a model to do it.