🔠 Tiktoken Tokenizer Info

Token counts are the invisible budget in every prompt. CLIP-era models truncate at 77 tokens and quietly drop the rest, so a lush description turns into a truncated one with no warning; 2026 LLM-encoded models removed the hard cap but kept a soft attention budget that people measure at roughly 75–100 effective tokens. Either way, "how long is this prompt really" is a question you cannot answer by looking at character count, and 🔠 Tiktoken Tokenizer Info answers it as a node.

It also does the thing you'd otherwise write a script for: chunk a long text into token-sized pieces, optionally snapped back to a word or sentence boundary so nothing is cut mid-word.

How it works

input_string gets tokenized with OpenAI's tiktoken. encoding_type gives you four choices: cl100k_base (the GPT-3.5/4 vocabulary, and the default), o200k_base (GPT-4o era), plus the gpt-4 and gpt-4o shorthand options that resolve to the same encodings by model name. The default is fine for budgeting - but that's the honest framing of what this node is: a proxy. CLIP has its own tokenizer, and a tiktoken count won't match a CLIP count exactly. Use it as a sanity check on prompt length, not as a hard limit calculator.

The outputs are unusually generous for a node this size. token_count, character_count and word_count are the headline numbers. Then split_string and split_token_ids give you the tokens in readable and numeric form, each with a _list variant for downstream nodes that want actual lists. special_tokens_used flags any special tokens in the text, and text_hash is a stable hash of the input - handy as a cache key or a filename component.

Then the three chunking outputs, which use token_chunk_size (default 75, and yes, that number is not a coincidence): token_chunk_by_size cuts on the token boundary and will happily bisect a word; token_chunk_by_size_to_word backs up to the nearest space so no token is lost and no word is split; token_chunk_by_size_to_section backs up further, to the nearest newline, period or comma. If you're splitting a long prompt into 77-token chunks to feed a chunked encoder, _to_word is the one you want.

It's flagged as an output node, so it executes and shows its numbers even if nothing is wired to it - that's how it doubles as a prompt-length readout you just leave in the graph.

Inputs and outputs

Inputs: input_string, encoding_type, and optional token_chunk_size. Thirteen outputs, listed above; the ones you'll actually use are token_count, token_chunk_by_size*, and text_hash.

Install

ComfyUI Manager → search "ComfyUI-mnemic-nodes" → install → restart. Or:

cd ComfyUI/custom_nodes
git clone https://github.com/MNeMoNiCuZ/ComfyUI-mnemic-nodes

The pack's requirements.txt includes tiktoken. One thing worth knowing: tiktoken downloads its BPE vocabulary files on first use (it caches them locally afterwards), so the very first run needs network access. On a fully offline box it can fail until the cache is populated.

Common issues

Nothing renders in the UI. It reports through the node's own display; there's no image or text output node involved. Check the node itself, not a preview.

Counts don't match your model's limit. Different tokenizer. This is a budget estimate.

First run hangs briefly. It's fetching the encoding file.

Chunks are splitting words. You're on token_chunk_by_size. Switch to _to_word or _to_section.