LLM (VLM) API

LLM (VLM) API is the point of this whole tiny pack: a single node that drops a real chat model into your ComfyUI graph and lets it read text, images, or both - and it sends the work to a remote API instead of hogging your VRAM. You point it at any OpenAI-compatible endpoint, wire in a picture, and it hands you back a text description you can feed to a prompt encoder, a captioning step, or just display. For a VLM-in-the-graph job, it's about as light as it gets: no model downloads, no GGUF files wrestling your 8GB card for the same memory your sampler wants.

The name does exactly what it says, by the way. It calls a real API. Unlike local-LLM captioners like JoyCaption or Florence-2, which ship weights and run on your GPU, this node is a client for somebody else's model. That's the whole trade spelled out: you give up local-and-free (and local-and-uncensored) and get frontier-chat quality without touching your VRAM. The same decision tree applies here as everywhere else - a local 8B is fine for rewriting prompts; an API is what you reach for when you want the big model and don't mind paying per call.

How it works

Under the hood it's the official OpenAI Python SDK, which is why any provider that speaks the OpenAI dialect works. Give it a base_url, an api_key, and a model name, and it builds a standard chat completion:

• Text you pass in becomes a text content block.
• An image (a ComfyUI IMAGE tensor straight from Load Image) is converted to PNG, base64-encoded, and sent as an image_url data URI - the standard OpenAI vision format. Send a real VLM (gpt-4o, a Qwen-VL, whatever your provider offers) and it will actually look at the picture.
• An optional system prompt is prepended as the system message.

Then it calls chat.completions.create and returns the model's reply.

Two quirks worth knowing before you wire it up. First, the node swallows errors: if the API call fails, the exception text is returned as your output string instead of crashing the graph. That keeps a bad key or a 429 from nuking your run, but it means "read the Show Text output" is your debugging loop. Second, you must supply something - leave both text and image empty and you get the string Error: No text or image provided. on the wire.

The inputs that matter

• base_url - the API endpoint. https://api.openai.com/v1 for OpenAI proper; swap in Groq's or DeepSeek's OpenAI-compatible endpoint, or http://localhost:11434/v1 for local Ollama/vLLM/LM Studio. Some providers need the /v1 suffix, some reject it - check their docs.
• api_key - your key. A local server usually accepts any dummy string.
• model - the model name as your provider spells it.
• temperature (0–2, default 0.7) and max_tokens (default 4096, capped at 8192 by the widget) - boring but they matter: crank temperature up for creative rewrites, keep it low for captioning.
• image / text / system prompt - the content. Both text and system prompt are forced input sockets (no inline typing), so you wire them from text nodes - which is exactly why this pack also ships LLM (VLM) Prompt.

The single output, response (a STRING), plugs into Show Text, or into any downstream text-consuming node.

Install

ComfyUI Manager (search "ComfyUI-LVLM-API"), or the old-fashioned way:

cd ComfyUI/custom_nodes
git clone https://github.com/iwindy910/ComfyUI-LVLM-API

Then install its dependencies and restart ComfyUI:

cd ComfyUI
.\python_embeded\python.exe -m pip install -r custom_nodes\ComfyUI-LVLM-API\requirements.txt

That's openai, numpy, and Pillow - and since ComfyUI already ships numpy and Pillow, the only real new dependency is the OpenAI SDK. No model files, no weights, nothing to download on first run.

Troubleshooting

• Every failure comes back as text. Response starts with Error: or a stack-ish message? That's your key being wrong, your base_url being wrong, or the provider rate-limiting you.
• Vision model required for images. Point a text-only model at an image and the provider will reject the request. Pick an actual VLM for vision jobs.
• Big images = big payloads. The node sends a full-res PNG as base64. If your provider complains about size, downscale the image before it hits this node.
• The key lives in the workflow. api_key is a widget value, so it's saved in plaintext inside your workflow JSON/PNG. Share a workflow and you share the key. Fine for a local server or a throwaway key; use a fresh one for anything paid.

And the usual LLM-node caveat: custom nodes run arbitrary Python with full OS access, and the "VLM API" category has literally shipped malware once before. This pack is small enough to read end to end - worth the two minutes before you run it.

The sibling LLM (VLM) Prompt node just holds text for this one; if you landed here first, you'll want both.