ComfyUI Node
CV VLM Generate
Answers a text prompt about an image with a decoder-only VLM of the PaliGemma shape: a model folder under models/llm holding three .onnx parts - a vision encoder (image -> image-feature tokens), a token embedding (prompt ids -> text embeddings) and a language model ([image features | text embeddings] -> logits), each found automatically inside the folder. Mirrors the opencv vlm_inference.py sample (paligemma2-3b-pt-224, prompt 'cap en\n' to caption). Greedy decoding; runs in the interruptible DNN worker. The vision encoder output is CONCATENATED before the prompt embeddings, PaliGemma-style - VLMs with a different fusion need the seq2seq node or their own subgraph.
CV VLM Generate
- image
- response
- new_tokens
◄model▾►
◄promptcap en
►
◄max_new_tokens32►
◄vision_fileauto (from model folder)►
◄embed_fileauto (from model folder)►
◄lm_fileauto (from model folder)►
◄tokenizerauto (from model folder)►
◄stop_token_ids1►
◄image_size224►
◄pixel_mean0.5►
◄pixel_std0.5►
◄enginenew graph►
Categoryimage/CV/dnn
Inputs (13)
| Name | Type | Default | Description |
|---|---|---|---|
| image | NPARRAY,IMAGE | The image to ask about (first frame of a batch). Resized to image_size x image_size. Accepts a ComfyUI IMAGE/MASK directly (frame 0 of a batch) or an NPARRAY. Arithmetic ops (add, multiply, etc.) process the full IMAGE batch when both inputs have the same batch size. | |
| model | COMBO | Model FOLDER under models/llm (e.g. opencv/paligemma2-3b-pt-224). Its vision / embedding / language parts and its tokenizer are found inside it - override any of them below if the layout is unusual. | |
| prompt | STRING | cap en | Task prompt ('cap en\n' = caption in English for PaliGemma; a question for instruct VLMs). |
| max_new_tokens | INT | 321–1024 | Maximum tokens to generate. NOTE: 'full sequence' mode re-runs the whole prefix every step - a 3B fp32 LM takes ~a minute per token on CPU, keep this small. |
| vision_fileopt | COMBO | auto (from model folder) | Override the vision encoder .onnx (input 'pixel_values', e.g. PaliGemma2's SigLIP vision_model.onnx -> (1, 256, 2304) image-feature tokens). 'auto' finds it in the model folder. |
| embed_fileopt | COMBO | auto (from model folder) | Override the token embedding .onnx (input 'input_ids' -> embedding rows, e.g. embedding.onnx). |
| lm_fileopt | COMBO | auto (from model folder) | Override the language model .onnx (input 'inputs_embeds' -> logits, e.g. gemma2_3b.onnx). |
| tokenizeropt | COMBO | auto (from model folder) | Override the tokenizer folder. 'auto' uses the model folder's own config.json + tokenizer.json (PaliGemma2: the gemma2 SentencePiece pair). |
| stop_token_idsopt | STRING | 1 | Comma-separated ids that end generation. PaliGemma/Gemma: 1 (<eos>). |
| image_sizeopt | INT | 22432–2048 | Square size the image is resized to for the vision encoder. |
| pixel_meanopt | STRING | 0.5 | Pixel normalization mean: one value or 'r, g, b' (SigLIP: 0.5). |
| pixel_stdopt | STRING | 0.5 | Pixel normalization std: one value or 'r, g, b' (SigLIP: 0.5). |
| engineopt | COMBO | new graph | DNN engine for cv2.dnn.readNetFromONNX. These graphs need the OpenCV 5 GRAPH engine. OpenCV 5.1 merged its two engines into one, so 'auto' is the graph engine there and this default follows the build. |
Outputs (2)
| Name | Type | Description |
|---|---|---|
| response | STRING | The generated answer (stop tokens removed). |
| new_tokens | INT | Number of tokens generated. |