LlamaVision Chat

This is the payoff node of the YALLM-LlamaVision pack. LlamaVision Model loads the weights; LlamaVision Chat actually talks. Give it a model, an image, and a prompt, and you get back text - a caption, a description, an answer about what's in the frame. It's how you make a ComfyUI workflow that sees.

Why you'd reach for it

The big use case is captioning. ComfyUI already has solid local taggers - Florence-2, JoyCaption - but they're terse. Llama 3.2 Vision writes full sentences and follows instructions: "describe the lighting and composition, mention the color of the jacket" works. That makes it a genuinely different tool for writing natural-language captions for a Flux or SDXL LoRA dataset. It's also handy for poking at your own renders - feed it a VAE-decoded image and ask what's wrong with the hands. The whole thing is local: no API key, no per-call billing, and it runs the actual Llama 3.2 Vision rather than a proxy.

The inputs that matter

• llm_model - wire in the output of LlamaVision Model. That's the one model source it accepts; it raises an error if you try to feed it an LLM from a different pack, because it needs the bundled vision processor.
• image - any IMAGE. Single image or a batch; if you batch, you get one completion per image (the output is a list).
• user_prompt - a multiline string, sent as the user turn of a one-shot chat with the image attached. Be specific if you want specific output.
• seed - sets the generation seed (constrained to 32 bits internally). Fixed seed plus low temperature equals repeatable captions, which is exactly what you want when captioning a dataset.
• llm_sampler (optional) - the LLMSAMPLER from this pack's LLM Sampler Settings node. Leave it disconnected and the model's own defaults (temperature 0.6, top-p 0.9) are used.

What comes out: completion, a STRING. Wire it into a Show Text node (the pack's sample workflow uses pythongosssss's Custom Scripts for that) or a Save Text node to write captions straight to disk.

How it works

Mechanically it's tidy. The node builds a multimodal message (image + text), runs it through Llama 3.2's chat template via the transformers processor, and generates up to 2048 new tokens - that cap is hardcoded, not configurable. When the last image in a batch is done it offloads the model to CPU (if it's an NF4 model), which frees VRAM for whatever your workflow wants to do next.

The honest footnotes

• The first run pays for the model download and load; subsequent runs are fast while the workflow keeps the model warm.
• VRAM is your constraint. NF4 runs in about 10GB and offloads after use; int8 or the BF16 original will eat your card and never give it back. Prefer the default NF4 model.
• If "LlamaVision" gives you a jolt: no, this isn't the ComfyUI_LLMVISION malware from 2024. Different author, different pack, small BSD-licensed codebase - read it if you want.

Installation

Same as the whole pack:

cd ComfyUI/custom_nodes
git clone https://github.com/asaddi/YALLM-LlamaVision
pip install -r YALLM-LlamaVision/requirements.txt

Or use ComfyUI Manager and search "YALLM-LlamaVision". The requirements will bump transformers to 4.46+, which can conflict with other nodes - the usual shared-environment caveat.

Troubleshooting

• "Only works with LlamaVision Model" error → you fed it an LLM from another pack. Use this pack's model node.
• Output truncated or too short → remember the 2048-token cap; for long captions, ask for less per image and batch.
• Nothing but VRAM errors → drop to the default NF4 quantized model.