Vision LLMs API Node (Old)

Vision LLMs API Node (the "(Old)" is the author's own label - a streaming sibling exists now) does one thing: it takes an IMAGE tensor from your graph, base64-encodes it, and sends it to any OpenAI-compatible vision model along with your prompt. The model reads the image and answers in text. That single capability covers a surprising amount of workflow: auto-captioning your outputs for posting or training, asking "what's wrong with this render?" as a quality check, or - the popular one - feeding a VLM an image and asking it to write the prompt that would recreate it, then routing that text into a sampler.

The default configuration is telling. api_url points at Alibaba's DashScope OpenAI-compatible endpoint (https://dashscope.aliyuncs.com/compatible-mode/v1) and model defaults to qwen-vl-max-0201. The pack's author is clearly a Qwen/DashScope user - the Alibaba Tongyi family that owns the local community's open models also runs the strongest cheap hosted VLMs - but the node is provider-agnostic. Point it at OpenAI's gpt-4o/vision models, a compatible Gemini gateway, or any OpenAI-dialect VLM endpoint.

How it works

The image path is worth knowing because it explains some behavior. The node takes your IMAGE tensor, converts the first frame of the batch to PIL, saves it to a temp PNG, and base64-encodes it into a data:image/jpeg;base64,... URI. That URI goes into a chat-completions request as an image_url content part alongside your text prompt. So: only the first image in a batch is seen (this is not a batch captioner), and it always converts to a temp file first. Temperature is hardcoded to 0.5. tenacity handles retries on rate limits, timeouts, and connection errors.

One thing you'll notice immediately: the use_jailbreak toggle. Flip it on and the node replaces your system prompt with a baked-in "Advanced Text-to-Image Prompt Generator" mega-prompt (a meticulous, structured prompt-engineering persona), and wraps the request in an uncensored "Cora" persona conversation. That's a NSFW-adjacent feature living in a captioning node. Your call whether it's useful or sketchy - but know it's exactly what it looks like: a guardrail-dropping prompt injection baked into the source.

The inputs that matter

• image (IMAGE) - the tensor to analyze. First frame only.
• prompt - your question or instruction ("describe this", "write a prompt for this image").
• system_prompt - your own system instruction, unless use_jailbreak is on (then it's ignored).
• model, api_url, api_key - defaults are DashScope/Qwen; swap to match your provider.
• seed, use_jailbreak.

Outputs: LLM ANSWERS (STRING) plus pass-throughs of seed, model, api_url, api_key.

Installing it

Part of AhBumm/ComfyUI_BillBum_Nodes. ComfyUI Manager → search "billbum", or:

cd ComfyUI/custom_nodes
git clone https://github.com/AhBumm/ComfyUI_BillBum_APIset_Nodes.git
cd ComfyUI_BillBum_APIset_Nodes && pip install -r requirements.txt

Restart ComfyUI. Deps: the pack's usual light set plus the OpenAI client - no models to download.

Where people get burned

• Default URL/API key mismatch. The DashScope default only works with a DashScope (Aliyun) key. Any other key → auth error.
• First frame only. Feed it a batch expecting per-image captions and you get one answer.
• Model names drift. qwen-vl-max-0201 is a dated snapshot; DashScope rolls model versions, so a "model not found" error usually means pick the current one on the Aliyun console.
• If you only need captioning and your VLM refuses system messages entirely, the pack also has "Non-System Prompt VLMs API Node" for exactly that case.