Qwen3VL (External llama.cpp)

The name says it all and then some: Qwen3VL (External llama.cpp) runs the vision language model outside ComfyUI entirely. You point it at a llama.cpp server you're already running, it base64-encodes your image, POSTs it to the server's OpenAI-compatible chat endpoint, and hands you back a caption. No model file lands in your ComfyUI folder, no GPU memory gets eaten inside the graph, and - the thing that surprises people - there's no API key anywhere in this. It's your own server, your own GPU, your own model.

Why reach for it? Dataset captioning, mostly. If you're building training captions or want prompt-style descriptions ("Stable Diffusion Prompt", "Danbooru tags") without juggling a half-dozen captioner nodes, this is a clean, free alternative to the hosted-API captioners. JoyCaption gets the spotlight in this pack, but Qwen3VL is the more flexible of the two - it's a general VLM, so the same server that captions your training set can also answer "what's in this image?" as a plain question.

How it works

The node builds a caption prompt from your choices, converts the incoming IMAGE tensor to a PNG, base64-encodes it, and sends it to llama_url as a chat-completion request with the image inline. It's pure requests - no ComfyUI-specific inference at all. The mechanism lives in your server, so you need one running:

llama-server.exe -m "Qwen3VL-7B-Instruct-Q5_K_M.gguf" -ngl 33 -c 4096 --port 58080 --image-min-tokens 2048

Grab a Qwen3VL GGUF (Q5_K_M is the sensible middle of the Q ladder - Q8 if it fits, per the gguf quality guidance) and a recent llama.cpp build. The --image-min-tokens 2048 flag matters: it reserves vision context for the image, and without it captioning gets flaky.

The inputs that matter

• caption_type - the preset that shapes the output: Detailed Description, Straightforward Caption, Stable Diffusion Prompt, Danbooru tags, Art Critique, Product Listing, Social Media Caption. The Danbooru one is handy if you train on booru-style tags.
• caption_length - any/very short/short/medium-length/long/very long, or a number from 20–260 to set an exact word budget.
• extra_options - free-form extra instructions appended to the prompt (default: "Use single-line output description.").
• temperature / top_p / max_tokens - standard sampling knobs. Keep temperature around 0.7; lower it if captions get rambly.
• llama_url - defaults to http://127.0.0.1:58080/v1/chat/completions. Change it if your server lives elsewhere.

Two outputs: caption (the generated text) and prompt (the exact prompt that was sent - useful when you're tuning extra_options and want to see what the model actually got).

Install

Install the pack via ComfyUI Manager (search "MiraSubPack") or:

cd ComfyUI/custom_nodes
git clone https://github.com/mirabarukaso/ComfyUI_MiraSubPack

Restart ComfyUI. The pack has no requirements.txt, so the captioning nodes only need requests, which ComfyUI ships. Everything heavy - the GGUF, the mmproj if you use JoyCaption - lives on the external server, not in the pack.

Troubleshooting

• "Connection failed" captions. The output text will literally start with Error: - check that llama.cpp is running, on the right port, and that llama_url matches (note the /v1/ in the Qwen3VL default vs the bare /chat/completions the JoyCaption nodes use).
• Garbage or truncated captions. Raise --image-min-tokens and make sure -c (context) is generous; a 2048-token image budget on top of a long instruction eats context fast.
• Slow batch runs. The node processes images one at a time and each waits the full request. That's the price of an external server; a faster model or smaller context window is the lever.

One caveat worth knowing: this is a satellite node in a "test nodes" pack from the author of ComfyUI_Mira, so polish is thin - there's no queue or parallelism, and errors come back as caption strings rather than ComfyUI exceptions. For a free, no-key local captioner, that's a fair trade.