PaLI-Gemma (Official Segmentation)

Most VLM nodes in this pack hand you a string. Paligemma is the odd one out - and it's worth a look precisely because of that. It wraps Google's PaliGemma 3B vision-language model and gives you three outputs: a text description, a MASK, and a visualization image. That means it's not just a captioner; flip the task type to Segmentation and it'll draw a mask around whatever you name. It's the only node in this pack that hands you something you can wire straight into an inpainting pipeline.

What it's for

• Captioning - the default task, and the default model is a fine-tune for exactly this job.
• Segmentation - name an object ("segment the dog", "find the red car") and get a mask back.
• Question Answering - ask a question about the image, get an answer.

That segmentation output is the differentiator. Text-grounded masking - point a VLM at the image, name the object, derive a mask - is exactly the pattern modern inpainting workflows use, and this node gives you the mask directly instead of a bounding box you have to convert.

How it works

It downloads whichever PaliGemma checkpoint you pick from Hugging Face (the model_id dropdown lists Google's official fine-tunes: cococap for captioning, refcoco-seg for segmentation, vqav2 for Q&A, and so on). The default gokaygokay/sd3-long-captioner-v2 is the author's own PaliGemma fine-tune for producing long, natural captions - a good default for SD3-family prompt work. For segmentation it layers a mask decoder on top of the model and returns the segmented region as a binary mask plus a colored overlay image. Note that PaliGemma is a 3B model with a license you must accept on Hugging Face before the download will work.

The inputs that matter

• model_id - which fine-tune. Match it to your task_type (cococap for captions, refcoco-seg for segmentation).
• task_type - Captioning / Segmentation / Question Answering.
• prompt - for segmentation this is the object description; for Q&A it's the question. The tooltip spells out the per-task phrasing.
• quantization - None / 8bit / 4bit for VRAM; 8bit is a sensible default if the full model won't fit.
• The segmentation family: fill_mask, mask_color, mask_opacity, mask_threshold, mask_blur - how the mask overlay looks and how hard the mask edges are.

Outputs: description (STRING), mask (MASK), visualization (IMAGE). Wire the mask into a masking/inpainting node, or just view the visualization.

Install

cd ComfyUI/custom_nodes
git clone https://github.com/gokayfem/ComfyUI_VLM_nodes
python -m pip install -r ComfyUI/custom_nodes/ComfyUI_VLM_nodes/requirements.txt

First run downloads the checkpoint (a few GB) - and remember the license acceptance on Hugging Face.

Gotchas

Don't run it on CPU and expect anything fast - it's a real 3B transformer. And for segmentation, the model is only as good as the fine-tune you picked: refcoco-seg wants object-y phrases, not full sentences. Start with "segment the X" phrasing, look at the mask, and adjust mask_blur if the edges are too hard.