🎯 LLMs Vision | 图像理解

This is the flagship of the ComfyUI-LLMs pack, and the node that makes the whole thing worth installing. You feed it an image and a prompt, it sends both to a cloud vision model, and it hands you back a text description. That's it - no local VLM, no gigabytes of weights, no VRAM bill. OpenAI's GPT-4V, Zhipu's GLM-4V, Alibaba's Qwen-VL, and Google's Gemini all behind one node, selected from dropdowns.

Why would you want that inside ComfyUI? The obvious use: captioning your own outputs. Take whatever the sampler just made, feed it back through this node, and get a natural-language description you can reuse as a prompt or a training caption. Combined with the pack's chat node you can close the loop - describe the image, then have the LLM turn that description into a better prompt for the next pass. It's a small, local version of the "critique the output and iterate" workflow that's been all over the diffusion communities lately.

How it works

The node takes your IMAGE tensor, converts it to a base64-encoded PNG, and dispatches to the right backend - the OpenAI-compatible path (works with any endpoint that speaks the vision chat format), Zhipu's SDK, Alibaba's DashScope MultiModalConversation, or Gemini's generate_content. Each path is imported lazily, so you only pay for the providers you've configured.

The model dropdowns come straight from settings.yaml, so the model names you see are the ones you listed. One footgun baked into the design: the model dropdown shows every model from every provider at once, and if you pick a model that doesn't belong to your selected model_type, the node doesn't crash - it returns an error string as its output. Confusing if you're not expecting it, and it's the single most common way to trip this node up.

The inputs that matter

• image - any IMAGE wire. Usually straight from a VAE Decode.
• model_type - openai, glm4, ali, or gemini. Pick the provider, then pick the model below.
• model - the specific vision model, e.g. gpt-4-vision-preview, glm-4v, qwen-vl-plus, or gemini-pro-vision. Remember: it must belong to the selected model_type.
• prompt - multiline. The default asks for a detailed description of subject, setting, lighting, mood - a solid captioning prompt. Swap it for "describe only the woman's face" or "write a style tag list" and the node becomes a different tool.

The single output is a STRING - wire it to a Show Text node, a CLIP Text Encode, or into the chat node as a user prompt.

Installing it

Same as the whole pack: ComfyUI Manager → search "ComfyUI-LLMs", or clone it manually.

cd ComfyUI/custom_nodes
git clone https://github.com/leoleelxh/ComfyUI-LLMs
cd ComfyUI-LLMs
pip install -r requirements.txt
cp settings.yaml.sample settings.yaml

Under vision_models you'll find an openai, glm4, ali, and gemini block - drop your API keys in, restart ComfyUI, and the node's dropdowns repopulate. Run python validate_config.py afterward; it prints exactly which providers are configured and warns about placeholder keys.

Where people get burned

• Placeholder keys. The sample config ships with sk-xxxxx / xxxxxx / AIxxxxxx per provider. Miss one and that specific provider fails with an auth error while the others keep working.
• Model/provider mismatch. Picking qwen-vl-plus while model_type is set to glm4 gives you a Chinese error string in the output, not a nice red node. Match the two.
• Proxies. This pack is bilingual for a reason - the author clearly runs it against both the Chinese APIs (DashScope, BigModel) and the Western ones, and "some models may require proxy access" is in the README for a reason. If a provider silently fails and the key is right, this is why.
• Rate limits. You're calling someone else's paid API every time you run. A batch of 16 images is 16 requests and 16 billable calls. Budget accordingly.

And the boring but important one: LLM-vision custom nodes have a genuinely notorious malware chapter (the 2024 ComfyUI_LLMVISION incident, which ended in a federal prosecution). That's a category problem, not a this-pack problem - but it's exactly why you install from the official leoleelxh/ComfyUI-LLMs repo and don't grab the first "vision" node you see in a search.