LLM_Tokenize

Before a language model can think about your prompt, it has to break it into tokens - the small pieces of text its vocabulary actually knows. LLM_Tokenize does that inside ComfyUI: you give it a string, it gives you the list of integer token IDs, and suddenly you can see your prompt the way the model sees it.

This is the pack's raw-materials node. It's not something you need for basic prompt-and-reply work - the Call LLM nodes tokenize internally and you never see the result. You reach for LLM_Tokenize when you want to work at the token level yourself: counting tokens to check a context budget, constructing a token sequence by hand, or feeding tokens into the pack's lower-level nodes (LLM_Eval, LLM_Sample, LLM_Detokenize). It's the input half of the tokenizer that LLM_Detokenize is the output half of.

Inputs and output

• LLM - the loaded model. Tokenization is model-specific, so this node uses your model's own tokenizer. Same text tokenizes differently on a Llama model than on a Qwen model.
• text - the string to tokenize. Multiline, so whole paragraphs are fine.
• add_bos (true) - whether to prepend the begin-of-sequence token. For a standalone prompt you usually want this on; it's the marker that says "a new sequence starts here." If you're concatenating chunks of pre-tokenized text, you may want it off to avoid inserting BOS mid-sequence.
• special (false) - whether to allow special tokens (like BOS/EOS) in the output. Off by default, because in normal tokenization those markers aren't produced from ordinary text.

The output is INT, and it's a list - the token IDs.

What you'll actually do with it

The most practical daily use is token counting: tokenize a prompt, and the length of the list tells you exactly how much context it'll consume. That's genuinely useful when you're near your n_ctx ceiling in Load LLM Model - LLM context is a real memory cost, and seeing your prompt as a list of integers makes "my prompt is 3,000 tokens" concrete instead of abstract.

Beyond that, you're in lab territory: LLM_Tokenize → LLM_Eval to push the tokens into the model's context, LLM_Sample to pick the next token, LLM_Detokenize to see the result. That pipeline is the pack's raw inference loop laid bare, and this node is its entry point.

One thing to keep in mind: tokens aren't words. Whitespace is often its own token, and a long word can split into several pieces. Don't expect the list length to match a word count - it won't, and that's correct behavior, not a bug.

Install

The pack's standard setup: ComfyUI Manager search "ComfyUI-Llama", or:

cd ComfyUI/custom_nodes
git clone https://github.com/daniel-lewis-ab/ComfyUI-Llama

with pip install llama-cpp-python on manual installs. GGUF models go in ComfyUI/custom_nodes/ComfyUI-Llama/models, restart, then Ctrl+F5 to hard-refresh. Everything lives under the LLM menu.

Bottom line: LLM_Tokenize is a quiet utility that earns its keep the first time you wonder "how big is my context, really?" - and it's mandatory if you want to explore the pack's token-level inference nodes. It won't generate anything for you, but it shows you exactly what will.