ComfyUI Node
CV Seq2Seq Generate
Generates text from an encoder-decoder model folder in models/llm. Two shapes: (a) image-captioning style - the encoder takes the image directly (input 'pixel_values' -> hidden states) and the decoder generates token ids against them ('input_ids' + 'encoder_hidden_states'); (b) Florence-2 style - a separate vision encoder produces image features that are concatenated with the embedded text prompt as the text encoder's 'inputs_embeds', and the decoder takes embedded start tokens ('inputs_embeds'). Pick a merged with-past decoder export for KV-cache. Greedy decoding; runs in the interruptible DNN worker.
CV Seq2Seq Generate
- image
- response
- new_tokens
◄model▾►
◄prompt►
◄start_token_ids50256►
◄max_new_tokens32►
◄encoder_fileauto (from model folder)►
◄decoder_fileauto (from model folder)►
◄vision_file[none]►
◄embed_file[none]►
◄tokenizerauto (from model folder)►
◄prompt_token_ids►
◄prompt_suffix_ids►
◄stop_token_ids50256►
◄image_size224►
◄pixel_mean0.5►
◄pixel_std0.5►
◄enginenew graph►
Categoryimage/CV/dnn
Inputs (17)
| Name | Type | Default | Description |
|---|---|---|---|
| image | NPARRAY,IMAGE | The image to describe (first frame of a batch). Resized to image_size x image_size. Accepts a ComfyUI IMAGE/MASK directly (frame 0 of a batch) or an NPARRAY. Arithmetic ops (add, multiply, etc.) process the full IMAGE batch when both inputs have the same batch size. | |
| model | COMBO | Model FOLDER under models/llm (e.g. opencv/vit-gpt2-image-captioning). Its encoder and decoder .onnx are found inside it - override them below if the layout is unusual. Note ViT-GPT2 ships no tokenizer of its own: set 'tokenizer' to the gpt2 folder. | |
| prompt | STRING | Text fed to the ENCODER (task instruction, e.g. Florence-2's '<CAPTION>' - BPE-tokenized as plain text exactly like HF does, add suffix '2' for the BART </s>). Ignored in shape (a). | |
| start_token_ids | STRING | 50256 | Comma-separated DECODER start ids (ViT-GPT2: 50256 = <|endoftext|> BOS; Florence-2: 2). |
| max_new_tokens | INT | 321–1024 | Maximum tokens to generate. |
| encoder_fileopt | COMBO | auto (from model folder) | Override the text/main encoder .onnx. Shape (a): input 'pixel_values' (ViT-GPT2's encoder_model.onnx). Shape (b): inputs 'inputs_embeds' + 'attention_mask' (Florence-2's encoder_model.onnx). |
| decoder_fileopt | COMBO | auto (from model folder) | Override the decoder .onnx (-> logits; inputs 'input_ids' or 'inputs_embeds' + 'encoder_hidden_states'). 'auto' prefers the merged with-past export, which is what KV-cache needs. |
| vision_fileopt | COMBO | [none] | [none] = shape (a): the image goes straight into the encoder. Otherwise a separate vision encoder .onnx ('pixel_values' -> image features, e.g. Florence-2's vision_encoder.onnx) whose output is prefixed to the embedded prompt; 'auto' finds it in the model folder. |
| embed_fileopt | COMBO | [none] | [none] = the decoder takes raw 'input_ids'. Otherwise a token-embedding .onnx ('input_ids' -> embedding rows, e.g. Florence-2's embed_tokens.onnx) used for the encoder prompt and the decoder steps. |
| tokenizeropt | COMBO | auto (from model folder) | Override the tokenizer folder. 'auto' uses the model folder's own config.json + tokenizer.json - ViT-GPT2 ships none, so point this at the gpt2 folder. |
| prompt_token_idsopt | STRING | Comma-separated ids fed to the encoder INSTEAD of encoding 'prompt' - for special/task tokens the OpenCV tokenizer cannot produce. Blank = encode 'prompt' as plain text. | |
| prompt_suffix_idsopt | STRING | Comma-separated ids APPENDED to the encoded prompt (BART-style tokenizers end the encoder input with </s>: use '2' for Florence-2). | |
| stop_token_idsopt | STRING | 50256 | Comma-separated ids that end generation (ViT-GPT2: 50256; Florence-2/BART-style: 2). |
| image_sizeopt | INT | 22432–2048 | Square size the image is resized to (ViT-GPT2: 224; Florence-2: 768). |
| pixel_meanopt | STRING | 0.5 | Pixel normalization mean: one value or 'r, g, b' (ViT-GPT2: 0.5; Florence-2: 0.485, 0.456, 0.406). |
| pixel_stdopt | STRING | 0.5 | Pixel normalization std: one value or 'r, g, b' (ViT-GPT2: 0.5; Florence-2: 0.229, 0.224, 0.225). |
| engineopt | COMBO | new graph | DNN engine for cv2.dnn.readNetFromONNX. These graphs need the OpenCV 5 GRAPH engine. OpenCV 5.1 merged its two engines into one, so 'auto' is the graph engine there and this default follows the build. |
Outputs (2)
| Name | Type | Description |
|---|---|---|
| response | STRING | The generated text (stop tokens removed). |
| new_tokens | INT | Number of tokens generated. |