🔠 Tiktoken Tokenizer Info

Tiktoken is OpenAI's own tokenizer library, and this node exposes it directly inside ComfyUI so you can see exactly how a piece of text gets chopped into tokens - the actual unit an LLM's context window and cost are measured in, which isn't the same as word count or character count and routinely surprises people the first time they check.

The practical reason to reach for this: anything token-limited downstream. This pack's own Groq LLM API node has a max_tokens field capping how much it generates, and if you're feeding it a long system_message or user_input, this node lets you check exactly how much of that budget your input text is already eating before you send it. It's equally useful for sanity-checking a prompt against any external API's context limit, or for splitting a long document into pieces that fit a fixed token budget for batch processing.

The two fields you'll actually touch: input_string is the text you're analyzing, and encoding_type picks which tokenizer scheme to measure it against - cl100k_base (the default) is the GPT-3.5/GPT-4-era encoding, o200k_base is the newer one behind GPT-4o, and gpt-4/gpt-4o are convenience aliases for those same two. This matters because token counts for identical text genuinely differ between encodings - pick whichever matches the actual model you're budgeting against, not just whatever's listed first. token_chunk_size (default 75) only matters if you're using the chunking outputs described below; ignore it if all you want is a simple count.

The count outputs are the straightforward payoff: token_count, character_count, and word_count give you three different ways to measure the same text, text_hash gives you a hash of the string (useful for deduplication or caching rather than anything cryptographic), and special_tokens_used reports which of the tokenizer's reserved special tokens, if any, show up in your text.

The chunking outputs are the more interesting half if you're splitting long text for batch processing: token_chunk_by_size cuts your text into pieces of roughly token_chunk_size tokens each, but a raw cut can slice straight through the middle of a word. token_chunk_by_size_to_word nudges each cut to the nearest complete word instead, so nothing gets truncated mid-token. token_chunk_by_size_to_section goes a step further and nudges to the nearest natural break - a newline, period, or comma - which is usually what you actually want if you're chunking prose rather than raw data. Each of these also ships a _list-suffixed twin (split_string_list, split_token_ids_list, special_tokens_used_list) that outputs the same data as a proper ComfyUI list rather than a single combined value, for anything downstream that wants to iterate item by item.

Installing it: ComfyUI Manager, search "ComfyUI-mnemic-nodes," or git clone https://github.com/MNeMoNiCuZ/ComfyUI-mnemic-nodes into custom_nodes, then restart. This one's worth flagging as low-risk specifically: it's a pure offline library doing local text analysis, no API calls, no key, nothing that can fail because a service is down.