Replicate lucataco/llama-3-vision-alpha

Llama 3 doesn't natively see images - Meta's original 8B/70B releases are text-only. llama-3-vision-alpha is a community project that bolts a vision encoder onto Llama 3 8B Instruct through a trained projection layer, giving the chat model eyes without retraining the whole thing from scratch. lucataco - one of the more prolific model-porters on Replicate, the guy who seems to package half the interesting open releases onto the platform within days - packaged it up and put it behind an API. This node calls that hosted version: image in, description out, no local weights involved.

It's a captioner, same job as any VLM node in this pack: describe a picture, answer a question about it, feed the result into a prompt or a caption file. What makes this one worth reaching for specifically is the base model underneath - Llama 3 is a strong instruction-follower, so the descriptions read more like natural writing than the terser output you get from some smaller VLMs.

How it works

Under the hood it's the standard adapter pattern: a vision encoder turns your image into embeddings, a projection layer maps those into Llama 3's token space, and the language model reasons over the combined sequence and writes a response. You never touch any of that - the node just sends your image and instruction to Replicate, their hardware runs the model, and you get a STRING back.

The inputs and outputs that matter

• image (required) - the picture to describe or ask about.
• prompt (default "Describe the image") - your instruction to the model, not an image-generation prompt. Swap it for a specific question ("what brand is this shoe?", "count the people in this photo") and you get answer-style output instead of a caption.

force_rerun (default off) forces a fresh generation instead of a cached one - useful if you want a different phrasing on the same image without changing the prompt. Output is a single STRING; wire it into a Show Text node, a prompt combiner, or the text input of whatever's downstream.

How to install it

One install covers every node in this pack.

• ComfyUI Manager: search ComfyUI-Replicate, install, restart.
• Manual:

  cd ComfyUI/custom_nodes
  git clone https://github.com/replicate/comfyui-replicate
  cd comfyui-replicate
  pip install -r requirements.txt
  

  then restart.

Nothing downloads to your machine - the model runs on Replicate's servers, so the only local dependency is the replicate Python client. Before you can run anything, set REPLICATE_API_TOKEN in the environment ComfyUI launches in: export REPLICATE_API_TOKEN="r8_..." (get one at replicate.com/account/api-tokens), then start ComfyUI from that same shell.

Common issues & troubleshooting

Node errors immediately. Almost always the token. REPLICATE_API_TOKEN has to be set before ComfyUI starts, in the exact environment it's running in. This trips people up especially on the portable Windows build - a global Windows environment variable isn't automatically picked up in portable mode, so you may need to set it in the launch .bat itself rather than just the Windows system settings.

It described the wrong thing, or missed detail. Vision-language models hallucinate, particularly on small text, exact counts, and fine detail. Ask a narrower question rather than "describe everything," and don't trust it blindly for anything that matters (label text, precise counts).

Same output every time. Replicate caches identical inputs; nothing new happens on a re-queue unless something changed. Flip force_rerun, or tweak the prompt.

It's slow or times out occasionally. That's a live API call with a cold-start possibility, not a bug in your graph - normal latency is a second or two, but the first call after idle time can take longer.

Support is thin if something breaks. This is an official Replicate-maintained pack, but response time on GitHub issues and Discord for individual model quirks tends to be slow - community threads are often faster than official channels for auth and integration problems.