Image to LLM Prompt

The whole ComfyUI-ImageToPrompt pack is this one node, and the whole node is a glorified API call. You feed it an image, it sends that image to any OpenAI-compatible vision API - GPT-4o in the cloud, or a local LLaVA server via llama.cpp or ollama - and it hands the model's text answer back as a STRING you can wire anywhere. That's it. No model files, no weights, no download that isn't already a pip install.

The name is a small lie, in a good way. It's not "turning images into prompts" in the sense of magic - it's an image-to-text bridge, and what you do with the text is up to you. In practice that usually is a prompt. The loop is an old community favorite: have a vision model caption the image, then feed that caption into a text encoder to regenerate or reimagine it. Back in 2024 the thread that popularized this (img2img in Ghibli style via LLaVA 1.6 writing the prompt string) claimed LLaVA captured "80%+ of important details," and caption-to-regenerate is still legit on LLM-encoded models, where the encoder speaks natural language natively. It's also the fastest way to get an alt-text description of a whole folder without standing up a full local captioning pipeline.

How it works

Mechanically it's about 70 lines. The node takes your image tensor (batch, height, width, channels - values 0–1), multiplies by 255, converts to a PNG in memory, and base64-encodes it. Then it builds a standard OpenAI chat.completions request: optional system message, then a user message whose content is your optional text plus the image as a data:image/png;base64,... URI. The response comes back and is returned as text. On any exception it doesn't crash - it returns the error as the text output and prints [ImageToLLMPrompt] ERROR: ... to the console. Convenient, since a bad API config shows up as readable text instead of a red node.

The inputs that matter

Three of them, and they're all required strings:

• base_url - your OpenAI-compatible endpoint, default http://localhost:8080/v1. That default is a llama.cpp server, so it assumes you already run one.
• api_key - default sk-test. Local servers ignore it, so sk-test is fine for them. Real cloud APIs need a real key.
• model - default gpt-4o, but swap to whatever your server serves (llava, qwen2-vl, etc.).

The optional system_prompt and user_prompt (both multiline) are where you actually steer the output - "You are an expert art critic" vs. "list only the tags." temperature (0–2, default 0.7) and max_tokens (default 2048) work exactly like you'd expect. The one output, text, is a STRING, so it plugs into any STRING input - a CLIP text encode, a prompt node, a file writer. Since the node is marked as an output node, the text also shows up in the UI.

Installing it

Via ComfyUI Manager, search "ComfyUI-ImageToPrompt" and install. Or manually:

cd ComfyUI/custom_nodes
git clone https://github.com/chrismrutherford/ComfyUI-ImageToPrompt
cd ComfyUI-ImageToPrompt
pip install -r requirements.txt

Then restart ComfyUI. The README's own manual-install snippet still has github.com/yourusername/... in the clone URL - a copy-paste artifact. Use the real repo above. Dependencies are just openai, pillow, and numpy (pillow and numpy you already have with ComfyUI), so no heavy installs and no model downloads - the model lives on the server you point at, not in your ComfyUI folder.

Where people get burned

• The defaults don't pair. gpt-4o + localhost:8080 + sk-test is three contradictory defaults. Pick one side: local LLaVA with model: llava, or cloud with a real key. The classic failure is leaving base_url at localhost while expecting GPT-4o - the API error comes back as text, and it'll say connection refused.
• Only the first image in a batch gets sent. The code takes image_tensor[0], so feeding it a batch from a Load Image Batch node silently ignores everything but the first frame. Feed images one at a time.
• Your image leaves your machine if you point at a cloud endpoint. That's inherent to the design, but worth saying plainly. Also: this pack installs the real openai package straight from PyPI, which matters because the one famous malware incident in this exact niche - ComfyUI_LLMVISION in 2024 - hid an infostealer in fake openai wheels. That ship has sailed here, but checking a node's requirements before pasting a real API key is a habit worth keeping.
• Empty or weird answers usually mean the model you picked doesn't support image inputs, not that the node is broken.

Is it the fanciest thing in the custom-node registry? No. But for "I want a vision model to read this image and tell me what to generate next," it's a straight line between your graph and the API, and it has no business being more complicated than that.