✨ Auto-LLM-Vision

Where the pack's text node writes, this one looks. Feed it an image, it describes what it sees, and that description gets appended to your positive prompt before CLIP encodes it. Same one-node philosophy as Auto-LLM-Text, different half of the brain - and the reason the pack's README demos work like "show it a zebra photo, generate a girl in a zebra-pattern dress." You don't write the details; a vision LLM extracts them from a reference and hands them to the sampler.

That's also the key to the pack's storyboard trick: describe the last frame, use the caption to prompt the next frame, and a series of images starts to cohere like a comic instead of five unrelated generations. The author literally frames it that way - "let LLM-Vision see the last-one-img for next-image."

How it works

The mechanism is the standard multimodal chat call. Your image_to_llm_vision IMAGE tensor gets converted to a PNG data URI (base64), then embedded as an image_url block inside the user message, alongside your llm_vision_ur_prompt text. That payload goes to the same OpenAI-compatible endpoint the text node uses - default http://localhost:1234/v1/chat/completions - and the reply is parsed from choices[0].message.content and appended to your base prompt.

Two things fall out of this. First, the model you load must actually be vision-capable - a plain text GGUF will answer with garbage or refuse. Second, this is the node where you notice the VRAM trade: the vision model the README suggests (llava-phi-3-mini) is around 7GB for the main file, plus a ~600MB mmproj adapter.

The inputs that matter

• image_to_llm_vision (IMAGE) - the reference image. Wire any Load Image / preview output here.
• llm_vision_ur_prompt - what you ask about the image. Default: "What's in this image?"; swap in "What pose is she in?" or "What is the man wearing?" to steer the caption.
• llm_vision_system_prompt - the default is a generic "describe an image" assistant; tighten it if you want a captioner's voice.
• llm_vision_max_token (50 default) and llm_vision_tempture (0.8 default) - note the temperature is higher than the text node's 0.3, so captions are more varied run to run.
• llm_vision_result_append_enabled - master switch for whether the caption lands in your prompt at all.
• llm_apiurl / llm_api_model_name - shared with the rest of the pack; point the model name at the vision model you loaded.

There's no llm_text_* group here - this node is vision-only, and it will happily report 🌀LLM-Text empty on the outputs.

Outputs and wiring

Same seven outputs as every node in the pack, but the interesting ones are:

• postive / negative (CONDITIONING) → sampler. The positive is your base prompt + the vision caption.
• 🌀LLM-Vision (STRING) - the raw caption, the thing you'll actually look at to sanity-check what the model thought it saw.
• 🌀postive+LLM-Text+LLM-Vision (STRING) - the assembled prompt text if you want it elsewhere.

Install

Same pack install as its siblings - Manager, search auto, install ComfyUI-decadetw-auto-prompt-llm, restart. Or:

cd ComfyUI/custom_nodes
git clone https://github.com/xlinx/ComfyUI-decadetw-auto-prompt-llm
# restart ComfyUI

The pack itself installs nothing heavy (requirements.txt is just numpy; it leans on requests and Pillow, which ComfyUI already has). The real setup is a vision-capable model in LM Studio or ollama. The README's picks: llava-phi-3-mini (the f16 main file plus its mmproj adapter), Llama-3.1-Unhinged-Vision-8B Q8, or - the budget choice for low VRAM - Eris_PrimeV4-Vision-32k-7B IQ3_XXS at ~2.8GB. With SDXL the README suggests 8GB+ VRAM.

Common issues

• Garbage or refusal from the model. You loaded a text-only model. Load a vision model, or your caption reads like a confused chatbot.
• [Auto-LLM][Result][Missing LLM-...] in the output. Same failure mode as the text node: unreachable server means the error string gets encoded into your conditioning instead of a loud crash. Check the server, the URL, and the model name.
• VRAM pressure. Vision GGs are bigger; between the checkpoint and the captioner, an 8GB card is tight. The IQ3_XXS quant exists precisely for this.
• It re-describes the same image forever. The node re-runs when its inputs change. If you want fresh captions from one static reference, this node won't invent new ones - that's what recursive mode and feeding the last output back in are for.
• The llm_*action_cmd fields run real shell commands (shell=True in the source) and the README flags the security risk itself. Don't paste in unknown commands, and don't expose a ComfyUI running this pack to the internet.