IW TokenizerVocab

IW TokenizerVocab dumps a CLIP model's tokenizer vocabulary to a JSON string. Feed it any CLIP and it returns the full word→token mapping the model actually uses - a window into how your prompt becomes numbers, which is a surprisingly educational thing to have.

Why this exists

Every prompt you type gets chopped into tokens by a tokenizer before the text encoder does anything with it. That vocabulary is where words live or die: "photorealistic" might be one token or five, and the difference matters for how strongly a model latches onto it. The KB's own research on embeddings found exactly this effect - how a tokenizer splits on underscores changes whether a trained trigger word survives (the bad_prompt_version2 name tokenizes differently than its literal text). TokenizerVocab lets you look at that mapping directly instead of guessing. Want to know if a phrase is a single token, or whether an embedding's trigger word exists in the base vocabulary at all? This is the tool.

The honest limit: SDXL only, and only CLIP-L

The README says it plainly: "limited to SDXL clips atm," and the source confirms it. It checks whether the CLIP's tokenizer is the SDXL dual-encoder type, and if so it pulls the vocabulary from the clip_l half (the SD 1.x-style small encoder, not the big OpenCLIP-G). If you hand it an SD 1.5 CLIP or anything newer - a T5, or a Qwen/LLM-based encoder from the current generation of models - the check fails and you get an empty object back. So think of this as an SDXL-era debugging/educational node, not a universal one.

What you get on a successful run is a big JSON mapping of token strings to IDs, as one STRING output. You'll usually want to pipe it into a SaveString (json mode) or a print node rather than eyeball it in the canvas - SDXL's CLIP-L vocab is tens of thousands of entries.

What you set

Just one input: clip - wire it from a Checkpoint Loader or CLIP Loader, or anywhere a CLIP flows through. One STRING output. That's the whole surface.

Installing

cd ComfyUI/custom_nodes
git clone https://github.com/iwanders/ComfyUI_nodes

Restart ComfyUI, or grab it via ComfyUI Manager (search iwanders). No dependencies, no model downloads - it reads the vocab from the CLIP you already loaded. If you're training embeddings or just curious why the model insists on splitting your favorite word, this is a genuinely handy peek under the hood.