PaliGemma 3B Captioner

The PaliGemma 3B Captioner runs Google's paligemma-3b-pt-224 entirely on your machine - free, private, no API key - and it's the most flexible of the captioning nodes in the ComfyUI_DW_Chat pack (yiwangsimple/ComfyUI_GroqChat). Beyond plain captioning, its task_prefix input lets you switch into detect and segment modes, which is more than most in-graph captioners bother with.

Reach for it when you're auto-captioning a batch - say, images you're about to use for training. The KB's lora-training essay notes that for LLM-encoder models the captions should be natural language rather than tags; this node's output is exactly that. It's the lighter-weight alternative to dedicated dataset tools like JoyCaption if you want to stay inside the graph.

How it works

The first run downloads the model from Hugging Face into ComfyUI/models/PaliGemmaCheckpoints/files_for_paligemma_3b_pt_224 (a few GB - plan for it). It then builds a prompt like caption en: caption, runs the processor, and samples the output with top_k - so captions vary run to run. After each generation it unloads the model from VRAM, unless you flip keep_alive on.

One thing to know: pt-224 is the pretrained base, not the instruction-tuned "mix" variant. PaliGemma's base was trained with task prefixes, which is why caption, detect, and segment work - but it is not a chat model, so don't type questions into the prompt field and expect a conversation. Use the prefixes and short task text.

Inputs and outputs

The inputs that matter:

• image - anything ComfyUI can feed as an IMAGE tensor.
• task_prefix - caption (default), detect, or segment.
• language - the model replies in en, es, fr, de, or zh.
• quantization - none / 8-bit / 4-bit for VRAM relief.
• keep_alive (optional) - true keeps the model resident between runs.

It returns caption (STRING) and seed (INT). The seed output feeds the standard ComfyUI seed loop with control_after_generate, which is a nice touch for batch captioning variety.

Installing it

Same as the rest of the pack:

cd ComfyUI/custom_nodes
git clone https://github.com/yiwangsimple/ComfyUI_GroqChat

Restart ComfyUI (or install via Manager, searching "ComfyUI_DW_Chat"). The pack's requirements cover transformers, torch, and huggingface_hub, so the node's imports resolve out of the box.

Common issues

The one that trips people: selecting 8-bit or 4-bit can fail with an import error, because that path needs the bitsandbytes package - and it is not in the pack's requirements.txt. If you want quantization, pip install bitsandbytes first.

Performance is the other tradeoff. With keep_alive off (the default) the 3B model reloads and unloads on every run - fine for a few images, miserable for captioning a thousand. Flip keep_alive on for batch jobs and accept the VRAM residency (roughly 6GB in fp16, less with 4-bit). If you're VRAM-starved, 4-bit + keep_alive is the sweet spot.