ComfyUI Node
CV DNN Tokenizer
Encodes text to token ids or decodes ids back to text with OpenCV 5's built-in tokenizer (cv2.dnn.Tokenizer: BPE for the gpt2/gpt4/qwen2.5 families, SentencePiece for Gemma). Lists every folder under models/llm holding an OpenCV-format config.json plus the model family's tokenizer.json (e.g. opencv_extra testdata/dnn/llm/<family>/). The 'CV LLM/VLM/Seq2Seq Generate' nodes tokenize internally - use this node to inspect tokens or to build prompts token by token.
CV DNN Tokenizer
- token_ids
- token_ids
- text
- count
◄mode▾►
◄tokenizer▾►
◄textWhat is OpenCV?►
Categoryimage/CV/dnn
Inputs (4)
| Name | Type | Default | Description |
|---|---|---|---|
| mode | COMBO | encode: 'text' -> token_ids. decode: the wired 'token_ids' -> text. | |
| tokenizer | COMBO | Model folder under models/llm holding the family's config.json + tokenizer.json (e.g. opencv/qwen2.5-0.5b-instruct). Relative to the model root, so the workflow stays portable. | |
| textopt | STRING | What is OpenCV? | Text to encode (encode mode). |
| token_idsopt | NPARRAY | (N,) or (1, N) integer token ids to decode (decode mode). |
Outputs (3)
| Name | Type | Description |
|---|---|---|
| token_ids | NPARRAY | (1, N) int32 token ids (encode mode; empty (1, 0) in decode mode). |
| text | STRING | Decoded text (decode mode; the input text echoed in encode mode). |
| count | INT | Number of tokens encoded / decoded. |