Qwen Token Analyzer

This is a read-only inspection node: paste in a prompt and it tells you how Qwen actually tokenizes it - the token count, the special tokens, the vision-token sequences, the whole breakdown. It generates no image. It's a debugging lens for the one component that quietly decides everything about how your words land: the text encoder.

Here's why that lens matters. The modern image models swapped CLIP for a general-purpose LLM encoder - Qwen2.5-VL for the Qwen-Image family, Qwen3 for Z-Image - and that single change retired a pile of old habits at once. The 77-token CLIP limit is gone; these encoders take thousands. But that's not license to write novels: from around 400 Z-Image generations one user found drift sets in past roughly 75–100 effective tokens, so the hard limit just became a soft one at almost the same length. The trouble is you can't feel token count by eye, especially once vision tokens like <|vision_start|><|image_pad|><|vision_end|> enter the picture and eat budget you didn't account for. This node makes that count visible.

How it works

You hand it text - including any special or vision tokens - and it runs it through the Qwen tokenizer and reports back. It counts the special tokens, finds the structured sequences (like the vision-token wrappers), and estimates the total token length. Turn on show_token_ids and it'll give you the raw ids too, which is where you go when you suspect the tokenizer is splitting a word or a tag in a way you didn't expect.

The inputs and outputs that matter

• input_text - the prompt to analyze. The default is pre-loaded with a vision-token example so you can see the format the node is built to dissect.
• show_token_ids (default on) - include the numeric token ids in the breakdown. Leave on when you're debugging; it's the most informative view.
• validate_coordinates - checks any spatial/coordinate tokens in the text for validity, relevant if you're experimenting with the pack's spatial-token features.

The useful outputs are token_breakdown and sequences_found (human-readable), plus two integers you can wire into logic elsewhere: total_special_tokens and estimated_total_tokens. There's also a full analysis_json if you want to parse it. This is an output node, so it displays its report inline.

How to install it

ComfyUI Manager → ComfyUI-QwenImageWanBridge → install → restart. Or:

cd ComfyUI/custom_nodes
git clone https://github.com/fblissjr/ComfyUI-QwenImageWanBridge

then restart. It only needs the tokenizer, so it's light - no diffusion model required to inspect a prompt.

Common issues & troubleshooting

My estimated token count seems high. Vision-token sequences and chat-template scaffolding add up fast. If you're running long past ~75–100 effective tokens on an LLM-encoded model, that's your adherence quietly degrading - trim to a few strong concepts, subject first.

The ids don't match another tool's count. Different tokenizers split text differently, and estimates are estimates. Use this node's number as a Qwen-specific reality check, not as gospel that transfers to a CLIP or T5 pipeline.

Analyzer or Debugger? This pack ships both, and they overlap - a sandbox thing. QwenTokenAnalyzer is the lighter "just tell me the counts and sequences" view; its sibling QwenTokenDebugger adds validation, template suggestions, and a corrected string output. If you want to understand, use this; if you want to fix a malformed prompt, use the Debugger.