Qwen Token Debugger

QwenTokenDebugger is the "do something about it" half of the pack's token tooling. Where QwenTokenAnalyzer reports what's in your prompt, the Debugger validates it, hands back a corrected string, and even suggests templates - so it's the one you reach for when a prompt with vision or spatial tokens is misbehaving and you want it fixed, not just measured. Both nodes exist because this is a research repo and the author built the tool twice as it evolved; if you only keep one for troubleshooting, keep this one.

The reason a "token debugger" is a sensible thing to want: on LLM-encoded models, the prompt isn't a bag of tags anymore - it's wrapped in a chat template and read like an instruction. Special-token sequences (the vision wrappers <|vision_start|>…<|vision_end|>, coordinate/grounding tokens) have to be well-formed or the encoder does something you didn't intend, and it fails silently rather than throwing an error. That silent failure is the trap. A malformed vision block doesn't crash; it just quietly produces a worse result, and you spend an hour blaming the sampler.

How it works

You feed it text, choose how deep to look with analysis_mode, and it parses the token stream, flags anything malformed, and produces a validated_text you can pass on. It counts special, vision, and spatial tokens separately - a more granular view than the Analyzer - and, with include_templates on, offers template_suggestions for how your prompt might be structured. There's an optional context_text input for when your prompt only makes sense alongside some surrounding context.

The inputs and outputs that matter

• text - the prompt under the microscope; defaults to a vision-token example.
• analysis_mode (full / tokens_only / sequences_only / errors_only) - how much to report. full while you're learning the node; errors_only once you just want to know "is anything broken?"
• validate_coordinates (default on) - sanity-checks spatial/coordinate tokens, the ones most likely to be malformed by hand.
• context_text (optional) - extra surrounding text to validate against.

The payoff outputs: analysis_report (what it found), validated_text (the cleaned-up prompt to feed your encoder), and template_suggestions. Plus three granular counts - total_special_tokens, vision_tokens, spatial_tokens - as integers.

How to install it

ComfyUI Manager → ComfyUI-QwenImageWanBridge → install → restart. Or:

cd ComfyUI/custom_nodes
git clone https://github.com/fblissjr/ComfyUI-QwenImageWanBridge

then restart. Tokenizer-only, so it's light - no diffusion weights needed to debug a string.

Common issues & troubleshooting

It flags errors I didn't know I had. That's the whole value - malformed vision or coordinate sequences don't error out on their own, they just degrade the result quietly. Take the validated_text output and use that downstream rather than your original.

The template suggestions feel generic. They're a starting scaffold, not a magic prompt. This pack revolves around experimenting with system prompts and templates, so treat the suggestions as one option to test, not a prescription.

Do I need this and the Analyzer both? No. Pick one for a given task. The Analyzer is a lighter read-out; the Debugger validates, repairs, and separates vision from spatial counts. For actually chasing down a broken prompt, the Debugger is the sharper tool - and as with everything in this sandbox repo, expect the exact outputs to shift between updates.