CV Seq2Seq Generate

An image goes in, text comes out. That's the node. It runs ONNX exports of encoder-decoder models - ViT-GPT2 for straight captioning, Florence-2 for the task-token style - through OpenCV's DNN module, in a graph, with no PyTorch VLM loader and no Ollama server.

The honest framing before the how: this is the heretic way to caption an image, and the pack's author says so about his own DNN examples. Core ComfyUI and the popular captioner nodes run these models natively in PyTorch on the GPU. This runs them through cv2.dnn, which in practice means CPU and one image per execution. If you want captions for a LoRA dataset, use JoyCaption or a Florence node and be done. If you want to understand what the pieces actually are - or you're already in a cv2 pipeline and don't want to drag in a second runtime - this node exists and it works.

The two shapes it supports

The description lays out two wiring shapes, and which one you're in decides half the inputs:

(a) Image-captioning style. The encoder takes the image directly - pixel_values in, hidden states out - and the decoder generates token ids against those hidden states. ViT-GPT2 is this shape. The prompt input is ignored here, because there's no text prompt to speak of.

(b) Florence-2 style. A separate vision encoder turns the image into features, those features get concatenated with the embedded text prompt as the text encoder's inputs_embeds, and the decoder takes embedded start tokens rather than raw ids. That's why there are vision_file and embed_file inputs at all: shape (b) needs an extra ONNX file for each.

Greedy decoding, in the pack's interruptible DNN worker - so a long generation doesn't lock the whole ComfyUI process with an uninterruptible C call.

The inputs that actually matter

• image - the picture. First frame of a batch; resized to a square image_size. IMAGE/MASK or NPARRAY.
• model - a folder under ComfyUI/models/llm, not a file. Its encoder and decoder .onnx are found inside it.
• prompt - the encoder's text: Florence-2's task token such as <CAPTION>, BPE-tokenized as plain text. Ignored in shape (a). Append 2 via prompt_suffix_ids for the BART </s>.
• start_token_ids / stop_token_ids - comma-separated ids. ViT-GPT2 uses 50256; Florence-2/BART uses 2. Wrong values here are the single most common reason you get one token, or an empty string, or a paragraph of rambling.
• max_new_tokens - 1 to 1024, default 32.

Then the shape-specific ones: vision_file and embed_file for Florence-2, tokenizer (auto uses the model folder's own config; ViT-GPT2 ships no tokenizer, so you point this at the gpt2 folder), image_size (224 for ViT-GPT2, 768 for Florence-2), and the normalization pair pixel_mean / pixel_std - Florence-2 wants 0.485, 0.456, 0.406 and 0.229, 0.224, 0.225, ViT-GPT2 wants 0.5 on both. Get those wrong and the caption is fluent nonsense, which is worse than an error.

engine defaults to the OpenCV 5 GRAPH engine, which these graphs need. Auto-encoder overrides - encoder_file, decoder_file - are there for unusual folder layouts; auto prefers the merged with-past decoder export, which is what KV-cache needs.

Outputs are response (the text, stop tokens stripped) and new_tokens. Wire response wherever a STRING goes - a CLIP text encode, a save-text node, an LLM rewriter in the same graph.

Install and models

ComfyUI Manager → ComfyUI CV → install → restart:

cd ComfyUI/custom_nodes
git clone https://github.com/bmad4ever/comfyui_cv
pip install "opencv-contrib-python-headless~=5.0.0.93"

Python ≥ 3.12, recent ComfyUI on the V3 node API. Models are not bundled. You need ONNX exports of the encoder and decoder (plus the vision encoder and embedding table for shape (b)) in ComfyUI/models/llm/<your-folder-name>, with a tokenizer.json alongside; model_sources.txt at the repo root has the URLs and licences. Note the pack's own caveat: a valid ONNX export is not guaranteed to be loadable here - modern architectures are limited by both cv2.dnn and the pinned OpenCV 5.0.0.93.

Common issues

• Empty output. Almost always the id triple: start_token_ids, stop_token_ids and prompt_suffix_ids must match the model family. The tooltips carry the right values for both families; copy them.
• One token and it stops. Stop token id is an id the model emits immediately, often because it's also the start token and you inverted which is which.
• Fluent garbage. Normalization. pixel_mean/pixel_std and image_size are per-model constants, not preferences. Florence-2 at 224 with ImageNet stats is the classic pair of mistakes.
• Nothing at all, or "graph engine" complaints. The default engine is deliberately the graph engine; these graphs don't run on the old one.
• It's slow, and that surprises you. It's cv2.dnn on CPU. That's the cost of doing it the OpenCV way, and it's the reason the pack's author tells you to use the core node when one exists.