Replicate yorickvp/llava-13b

LLaVA is the workhorse open vision-language model - you hand it an image and a question, it hands back text. For ComfyUI people that means one thing above all: captioning. Feed it a picture, get a description you can drop into a text-to-image prompt, use to auto-caption a training set, or just read back to understand what a model actually produced. This is the 13B version (yorickvp's packaging) run through Replicate, so you get a capable VLM with no local model to load: REPLICATE_API_TOKEN, pay-per-run, nothing downloaded.

LLaVA has been the default answer to "how do I caption images in ComfyUI" for a long time. It's general-purpose, reliable enough, and big enough (13B) to give genuinely useful descriptions rather than one-liners.

How it works

LLaVA bolts a vision encoder onto a language model, so it can "see" an image and talk about it. The node sends your image plus a text instruction to the hosted model on Replicate; the model reasons over both and returns a STRING. Crucially, the prompt here is your instruction to the model ("describe this," "what color is the car," "list the objects"), not an image-generation prompt.

The inputs and outputs that matter

• image (required) - the picture to look at.
• prompt (required) - what you want to know. "Describe this image in detail" for captioning; ask a specific question for VQA.
• temperature (0.2) - keep it low for factual, grounded descriptions. Higher makes it more creative and more likely to embellish, which is usually the wrong direction for a caption.
• max_tokens (1024) - cap on how long the answer gets. Lower it if you want terse captions.

top_p (1) rounds it out; force_rerun forces a fresh run. Output is a single STRING - wire it into whatever consumes text (a Show Text node, or the text input of another node).

How to install it

One install covers every Replicate node.

• ComfyUI Manager: search ComfyUI-Replicate, install, restart.
• Manual:

  cd ComfyUI/custom_nodes
  git clone https://github.com/replicate/comfyui-replicate
  cd comfyui-replicate
  pip install -r requirements.txt
  

  then restart.

Nothing lands on your GPU - LLaVA runs on Replicate's servers, so the only install is the replicate Python client. Set your token before launching ComfyUI: export REPLICATE_API_TOKEN="r8_..." (replicate.com/account/api-tokens). No token, no runs.

Common issues & troubleshooting

Auth error first run. REPLICATE_API_TOKEN must be set before ComfyUI launches. Set it, restart.

It output text, not an image - where's my picture? That's expected. This is a captioner: image in, STRING out. Feed that string into a text node or a downstream prompt input.

The description is wrong / made-up. VLMs hallucinate, especially about small details, counts, and text-in-image. Lower temperature toward 0 for the most grounded answer, ask a narrower question, and don't fully trust fine details - verify anything that matters.

Answers are too long or too short. That's max_tokens. Turn it down for tight captions, up for detailed ones.

Same answer on re-run. Replicate cached identical inputs. Change the prompt or set force_rerun. And remember each call is a billed Replicate prediction.