Generate Text with Llama Vision

Generate Text with Llama Vision runs a Llama 3.2 11B Vision model on a list of images and writes the answer as a string. In the pack's own words, this is the one you want when the job involves reading text off an image - the nf4 Pixtral build degrades badly on OCR and text-heavy images, while Llama Vision handles it better. For a beginner who just wants a captioner, this node with the default prompt ("Caption this image.") is the least fiddly start in the pack.

How it works

The node builds a proper chat template: an optional system prompt, a user turn with the image tags, and an assistant turn. Images go in as <|image|> tokens, and here's the catch the README is blunt about: the tags have to be before the instruction and consecutive for the model's vision attention to even engage. Unlike Pixtral, you can't scatter images through the prompt - they all pile up at the front. It does support multiple images in one generation, but the author found multi-image understanding unreliable: given two images in one message, the model mixes up first/second, left/right, and the colors between them. Single-image work is where this node shines.

Inputs that matter

• llama_vision_model - a VISION_MODEL from Load Llama Vision Model.
• system_prompt - optional (default empty). A real system prompt, unlike Pixtral's instruct-only setup.
• prompt - default "Caption this image."
• images (optional) - an IMAGE input. A list for one multi-image generation, not a batch; no batched text generation here.
• max_new_tokens - default 256.
• temperature / top_p / top_k - default 0.3 / 0.9 / 40. Low temperature = stable, consistent captions.
• stop_strings - default <|eot_id|>.
• seed, include_prompt_in_output, unload_after_generate - the usual.

Notice what's not here: no repetition_penalty. The author pulled it because including it crashed the CUDA kernel - a real "the code told me" moment, and worth knowing so you don't wonder where it went. Output is a single STRING.

Install

One-time pack setup - Manager search ComfyUI-PixtralLlamaVision, or clone the repo. Requirements: transformers >= 4.45.0, accelerate, bitsandbytes, torchvision. Model in ComfyUI/models/LLM (the nf4 quant from the README's HuggingFace links is the practical choice for normal VRAM).

Troubleshooting

• Model gives unrelated/mixed-up answers on two images - that's a known limitation, not a bug in your workflow. Split into single-image runs or switch to Pixtral for comparisons.
• Caption loops or repeats - no repetition penalty exists to turn down, so shorten max_new_tokens and rely on the stop string.
• OOM - 11B is heavy; use the nf4 quant and unload_after_generate.

Wire the STRING output into a save-text node and you have a perfectly respectable dataset captioner - especially when those captions contain text the image is showing.