Token Inspector (CCN)

"Why does my prompt only half-work?" is usually a tokenization question wearing a costume. Token Inspector (CCN) is the debug tool that answers it: give it a CLIP model and a prompt, and it dumps the full tokenization - every encoder, every chunk, every token with its ID and weight - as a readable string. It's the X-ray you run when a prompt is behaving strangely and you want to see what the model actually received, not what you typed.

This is where the weirdness lives. A word you think of as one thing gets split into pieces or merged into a neighbor. A rare term eats multiple token slots. Attention weights like (word:1.3) show up as weight annotations on individual tokens - and you can see in the dump why the classic A1111 weighting syntax either works or does nothing depending on the model, exactly the CLIP-versus-LLM split that prompt-engineering.md walks through. Once you've seen the token IDs, a lot of "why is this prompt ignoring this word" stops being mysterious.

How it works

The node calls the CLIP model's tokenizer and formats the result. For each encoder it prints every chunk, and inside each chunk every token position with its ID and weight (weights other than 1.0 are flagged, so you can see where emphasis actually landed). At the end it also computes a content-token count per encoder - tokens minus BOS/EOS - which is the number Token Counter (CCN) reports as its headline figure.

The output is a single STRING, shown on the node and logged to the console. Because it's an output node (OUTPUT_NODE), it always runs and displays - you can't accidentally skip it by not connecting its output to anything.

The inputs

• clip - the CLIP model whose tokenizer you want to interrogate.
• text - the prompt to tokenize. That's all there is.

Output: one STRING with the full breakdown. It's a diagnostic display, not a pipeline node - you're reading it, not wiring it forward.

How to install it

One pack, one install: ComfyUI Manager → search ComfyCollectorNodes → Install, or:

cd ComfyUI/custom_nodes
git clone https://github.com/valkymaera/ComfyCollectorNodes.git

Restart ComfyUI. No pip step, no extra dependencies.

Common issues

The main thing to get wrong here is expecting it to be useful on models that don't tokenize the way you assume. On a CLIP-encoder model the dump is gold - you'll see the 77-token chunking and can diagnose truncation directly. On an LLM-encoded model the tokenizer output is a chat-style instruction stream, and "weight" annotations mostly don't exist because the model doesn't use them; the dump is still honest, it just stops explaining what you hoped it would.

Also note the node reads clip.tokenize output as (token_id, weight) pairs - the format ComfyUI's core tokenizers produce. Exotic tokenizer wrappers may return extra per-token data, and the inspector is built to handle the standard case. If the output looks garbled, that's the first thing to suspect.