Text To Tokens (Shinsplat)

Somewhere between "my prompt is a text box" and "my prompt is a tensor" there's a step most people never see: the tokenizer splitting words into numeric ids. This node is a window into that step. It takes your text and a clip, runs the tokenizer, and hands you the result as readable data - plus a plain _prompt output for when you just want the cleaned text. Its real job, though, is feeding the pack's Clip Tokens Encode node so you can weight individual tokens instead of whole words.

How it works

Wire in a clip and type a prompt; the node runs clip.tokenize() and converts the result into a structured string where each token is a small dictionary:

{"word": "un", "token": 569, "weight": 1.0, "clip": "g", "index": 0},
{"word": "rel", "token": 1825, "weight": 1.0, "clip": "g", "index": 1},
{"word": "enting</w>", "token": 20526, "weight": 1.0, "clip": "g", "index": 2},

The word key is reference only; token is the numeric id; weight is what you'll tweak; clip says which stream it belongs to; index is the position in the list, useful if you're editing the data programmatically. The README's point: "unrelenting" becomes three tokens, and you can weight each of those separately - something a normal (word:1.3) prompt can't express. The before_ and after_ inputs sandwich extra text around the main box (and END is honored in all three). The author's tokenizer also restructures the back-end token data - he's openly of the opinion that the stock tokenizer output "gets worse with each implementation" - so the encoded result can differ from stock ComfyUI on the same seed.

Outputs: _prompt (the plain text, minus directives) and _tokens (the structured stream to pipe into Clip Tokens Encode).

How to install it

Part of ComfyUI-Shinsplat:

cd ComfyUI/custom_nodes
git clone https://github.com/Shinsplat/ComfyUI-Shinsplat

or ComfyUI Manager → "ComfyUI-Shinsplat" → restart. No extra deps.

Common issues

• It works with SD3, not just SD/SDXL - the author says the plain version is SD3-compatible and backwards-compatible, with a dedicated SD3 variant available for extra outputs. If the dedicated one fails on your setup, the plain one is the fallback.
• _tokens is text, not a tensor - correct; it's meant to be piped into Clip Tokens Encode, which converts it. Trying to read it as a tensor is the number-one confusion here.
• Token ids changed after a model swap - regenerate tokens with the new clip wired in.