Replicate lucataco/paligemma-3b-pt-224

PaliGemma is Google's compact vision-language model - a SigLIP vision encoder paired with a Gemma 2B language model - and this specific node is the pt-224 checkpoint: the raw pretrained weights at 224×224 resolution, not the instruction-tuned "mix" version. That distinction matters more than it sounds like it should, because it changes how you have to talk to it. lucataco packaged this one for Replicate; the node just calls the hosted model.

Here's the thing a beginner won't expect: a pt checkpoint doesn't chat. It was trained to complete task-prefixed strings, not to follow open-ended instructions. Look at the node's own default prompt - "caption es" - and that tells you everything. That's PaliGemma's task-prefix syntax for "caption this image, in Spanish." Other real prefixes from the same convention: "caption en" for an English caption, "answer en {question}" for VQA, "detect {object}" for detection, "ocr" to read text in the image. Type something conversational like "what's happening in this picture?" and you'll get worse results than typing the prefix syntax it was actually trained on.

How it works

You send an image plus a task-prefixed prompt string; the model encodes the image, conditions on your prefix, and generates a short text completion. It's not reasoning about your instruction the way a chat-tuned model would - it's pattern-matching to the task format it saw during pretraining. That's also why outputs tend to be short and literal rather than conversational.

The inputs and outputs that matter

• image (required) - the picture to caption or query.
• prompt (default "caption es") - this is where the task-prefix syntax lives. Swap es for another language code to caption in a different language, or switch the whole prefix to ocr / answer en <question> / detect <thing> for a different task.

force_rerun bypasses the cache for a fresh run. Output is a single STRING.

How to install it

One install covers the whole pack.

• ComfyUI Manager: search ComfyUI-Replicate, install, restart.
• Manual:

  cd ComfyUI/custom_nodes
  git clone https://github.com/replicate/comfyui-replicate
  cd comfyui-replicate
  pip install -r requirements.txt
  

  then restart.

Nothing downloads locally - PaliGemma runs on Replicate's servers. Set REPLICATE_API_TOKEN (from replicate.com/account/api-tokens) in the shell before launching ComfyUI, or the node has nothing to authenticate with.

Common issues & troubleshooting

Output is a single unhelpful word, or ignores what you asked. This is the number one trap with this specific checkpoint: typing a natural-language question instead of a task prefix. If you want it to actually answer a question, the prompt needs the answer en prefix, not a bare sentence. Getting this wrong looks like a broken model; it's actually a prompt-format mismatch.

Auth error on first run. REPLICATE_API_TOKEN isn't set in ComfyUI's environment. On the portable Windows build especially, a global environment variable set through Windows settings doesn't always get picked up - set it in the launch script directly if export/set before running doesn't stick.

You wanted a chatty description and got something terse. That's the pt checkpoint by design - it's the pretrained base, tuned for short task completions, not conversation. If you consistently want instruction-following behavior without prefix syntax, that's a different PaliGemma checkpoint (the fine-tuned "mix" release), not this one.

Same result on re-run. Cached. Change the prompt or flip force_rerun.

Cost adds up on batches. Every call is a billed Replicate prediction - fine for occasional use, worth budgeting for if you're running this over a folder of images.