SDXL Token Analyzer

Prompts don't fail loudly. They get truncated at a token boundary you can't see, and you just get a worse image and no idea why. SDXL Token Analyzer is the diagnostic for that: it tokenizes your prompt through both SDXL tokenizers - clip_g and clip_l - segment by segment, and shows you exactly what the encoder will see and how many tokens each segment costs.

Why you'd reach for it

SDXL has two text encoders, and they tokenize differently. A prompt can fit comfortably in one and blow past the 75-token usable limit in the other - which means "it fits" isn't a single number, it's two. When you're writing long tag prompts, or debugging why the back half of a prompt seems to have no effect, this node is the fastest way to see the truth: per-segment token counts for both tokenizers, plus the decoded tokens so you can spot what the tokenizer actually glued together.

It's the natural partner to the pack's SDXLAutoBreak: run the analyzer, see where you're over budget, and let Auto Break fix it automatically.

How it works

The node takes your CLIP and text, splits the text on BREAK first, then tokenizes each segment on its own. That per-segment split matters - it avoids CLIP's internal 77-token chunking, so you see the real count per logical segment rather than what the raw chunking does. It strips start/end/pad tokens from the count, untokenizes the rest back into readable strings, and flags the details you care about:

• Tokens with a weight other than 1 render as (token:weight), so you can see emphasis at the token level.
• Embeddings render as [emb] placeholders, since they're tensors, not text.

Inputs and outputs

• clip - the SDXL CLIP to analyze with. Must expose clip_g and clip_l.
• text - the prompt to inspect.

Four outputs, all strings:

• g_tokens / l_tokens - the decoded token lists per segment, for the G and L tokenizers (segments separated by blank lines).
• g_token_count / l_token_count - comma-separated token counts per segment.

Wire the token outputs into a text preview node (Show Text / anything that renders a string) to actually read them.

Installation

Part of ComfyUI-Alchemine-Pack. Install through ComfyUI Manager (search "Alchemine") and restart, or:

cd ComfyUI/custom_nodes
git clone https://github.com/alchemine/comfyui-alchemine-pack
pip install -r requirements.txt

Restart ComfyUI. Pack dependency story is minimal - python-dotenv only - and these prompt nodes need no models or keys.

Common issues

The big one is scope: this is SDXL-only. No clip_g/clip_l on the CLIP you feed it and it errors. It's also a read-only analysis tool - it doesn't truncate or fix anything, it just tells you the damage. The mental model that helps: a segment's usable budget is 75 tokens (77 minus start/end), and since the g and l tokenizers disagree on counts, trust whichever is larger when deciding whether to add a BREAK. If you're feeding this from an SDXL workflow and the counts look higher than you expected, remember they're per segment and comma-separated - a two-segment prompt gives you two numbers.