ComfyUI Node
CV LLM Generate
Generates text with a decoder-only LLM loaded by OpenCV 5's new DNN engine (cv2.dnn.Tokenizer + KV-cache): Qwen2.5/Gemma3 instruct exports or GPT-2, as a model folder under models/llm (e.g. optimum-cli export onnx --task causal-lm-with-past, or the merged decoder of a transformers.js export). The chat template, stop ids and decode mode are dropdown choices - nothing is baked to one model. Greedy decoding (deterministic). Mirror of the opencv samples qwen/gemma3/gpt2_inference.py. Generation runs in the interruptible DNN worker (tokenization + decoding included).
CV LLM Generate
- response
- full_text
- new_tokens
◄model▾►
◄templatechatml (Qwen2.5)►
◄promptWhat is OpenCV?►
◄max_new_tokens32►
◄decode_mode▾►
◄lm_fileauto (from model folder)►
◄tokenizerauto (from model folder)►
◄stop_token_ids►
◄bos_token_id-1►
◄enginenew graph►
Categoryimage/CV/dnn
Inputs (10)
| Name | Type | Default | Description |
|---|---|---|---|
| model | COMBO | Model FOLDER under models/llm (e.g. opencv/qwen2.5-0.5b-instruct). Its .onnx and its tokenizer are found inside it; override either below if the layout is unusual. KV-cache mode needs a with-past/merged export; a plain causal-lm export only works in 'full sequence' mode. | |
| template | COMBO | chatml (Qwen2.5) | Prompt wrapper. chatml: Qwen2.5-Instruct (<|im_start|>user ...). gemma3: Gemma 3 instruct (<start_of_turn>user ..., prepends BOS). 'none': raw completion (GPT-2). The template also sets the default stop/BOS ids. |
| prompt | STRING | What is OpenCV? | User prompt, wrapped by 'template'. |
| max_new_tokens | INT | 321–2048 | Maximum tokens to generate. CPU inference is roughly a few tokens/second for a 0.5B model - keep this small. |
| decode_mode | COMBO | kv-cache: fast, one token per step through the graph, needs a causal-lm-with-past / merged export. full sequence: re-feeds the whole growing sequence every step (slow, but works with plain exports). | |
| lm_fileopt | COMBO | auto (from model folder) | Override which .onnx in the model folder is the language model. 'auto' picks the merged/only decoder export - set this only when the folder holds several and auto cannot tell. |
| tokenizeropt | COMBO | auto (from model folder) | Override the tokenizer folder. 'auto' uses the model folder's own config.json + tokenizer.json (or the nearest parent holding them); set this for an export that ships no tokenizer of its own. |
| stop_token_idsopt | STRING | Comma-separated ids that end generation. Blank = the template default (Qwen2.5: 151645, 151643; Gemma3: 1, 106; GPT-2: 50256 suggested). | |
| bos_token_idopt | INT | -1-1–1000000 | Token prepended to the encoded prompt. -1 = the template default (Gemma3 prepends 2; the others none). |
| engineopt | COMBO | new graph | DNN engine for cv2.dnn.readNetFromONNX. LLM ONNX graphs need the OpenCV 5 GRAPH engine; the classic 4.x engine cannot parse them. OpenCV 5.1 merged the two into one engine, so 'auto' is the graph engine there and this default follows the build. |
Outputs (3)
| Name | Type | Description |
|---|---|---|
| response | STRING | The generated text only (prompt stripped, stop tokens removed). |
| full_text | STRING | Prompt + response, detokenized. |
| new_tokens | INT | Number of tokens generated (before stop-id removal). |