Replicate andreasjansson/blip-2

BLIP-2 is an image-understanding model that does two related jobs: it can write a caption for an image, or it can answer a question about an image ("What color is the car?", "How many people are there?"). It's the model that popularized cheap, decent visual question answering before the big multimodal LLMs showed up. This node lets you call it from ComfyUI through ComfyUI-Replicate, so you can pull a text description out of an image mid-workflow.

Runs on Replicate's cloud, not your GPU - no local model to install.

How it works

BLIP-2's trick is efficiency. Instead of training a giant vision-language model from scratch, it bolts a small trainable module (a "Q-Former") between a frozen image encoder and a frozen language model, so it inherits a strong LLM's language ability without paying to retrain it. The upshot for you: give it an image and it either describes it or answers your question. The node sends the image to Replicate and returns the answer as a STRING.

Inputs and outputs that matter

Required input is an IMAGE. Output is a single STRING.

The behavior hinges on one toggle:

• caption (default false) - turn this on to get a plain caption of the image. Leave it off to run in question-answering mode.
• question - in QA mode, what you're asking. Default is "What is this a picture of?".
• context - optional prior context for the question, useful for follow-up-style queries.

use_nucleus_sampling and temperature (0.5–1.0) control how varied the answer is - nucleus sampling on with higher temperature gives more diverse, less deterministic replies; off gives you the straightest answer. force_rerun forces a fresh pass on an unchanged image.

How to install it

ComfyUI Manager: search ComfyUI-Replicate, install, restart. Manual:

cd ComfyUI/custom_nodes
git clone https://github.com/replicate/comfyui-replicate
cd comfyui-replicate
pip install -r requirements.txt

Restart, and set your token first:

export REPLICATE_API_TOKEN="r8_************"; python main.py

Token: replicate.com/account/api-tokens.

Common issues

Good news: BLIP-2 returns plain text, which is the "simple output" case the pack's README says it fully supports - no audio/JSON weirdness here.

The one behavior that trips people is the caption toggle. If you're getting terse "What is this a picture of?" style answers when you wanted a description, flip caption on. If you're getting a generic caption when you wanted an answer to a specific question, flip it off and fill in question.

Beyond that: a missing REPLICATE_API_TOKEN is the usual first-run error, and every call bills on Replicate. Worth knowing the model's age, too - BLIP-2's answers are short and literal. If you need richer, more reasoned descriptions of an image, the GLM-4V node in this same pack is a newer vision-language model that handles nuance better; BLIP-2 is the fast, cheap, good-enough option for a quick caption.