Vocab Encode

Models don't read words; they read numbers. Vocab Encode is the translator: give it a vocabulary and a comma-separated list of tokens, and it returns the index of each token as a tensor. "the,quick,brown" becomes [7, 42, 3] - whatever indices your vocab assigned. It's the d2l Vocab.__getitem__ as a node, and it's where text officially becomes math in this pack's pipeline.

The one behavior that matters most is the fallback: any token not in the vocabulary maps to the <unk> index instead of erroring. That's deliberate, and it's the safety net that keeps a sentence with one unseen word from killing your whole queue. But it's also the thing that bites people silently - you can feed a vocab built on one corpus text from another, and a chunk of your words quietly become <unk> with no error, no warning, no red node. The vocab size and your encode results are the diagnostic; if your indices look suspiciously repetitive, that's unseen-token collapse doing its quiet thing.

Inputs

• vocab - the cdlVocab object, straight from Vocab Build. Without it there's no table to look up against.
• tokens - a multiline STRING of comma-separated tokens, e.g. the,quick,brown. The format matches what Tokenize and Vocab Decode speak, so the chain flows.

Output

One output: indices, a cdlTensor of dtype long. That dtype matters - it's an integer-index tensor, ready for embedding layers, and unlike some pack nodes it doesn't silently coerce to float. This is exactly the kind of tensor Truncate/Pad expects, so the natural pipeline is Tokenize → Vocab Build → Vocab Encode → Truncate/Pad → model.

Installing it

Part of ComfyDL. ComfyUI Manager, search "ComfyDL". Or:

cd ComfyUI/custom_nodes
git clone https://github.com/Cynthia-lxx/ComfyDL
pip install -r ./ComfyDL/requirements.txt

Restart ComfyUI. Only matplotlib as a pack dependency; nothing to download.

Gotchas

The <unk> fallback is both the feature and the trap, so here's the honest framing: it's great when you're testing, and it's a landmine when you're not looking. If you encode text with a vocab that doesn't contain your tokens, you get a tensor full of the same index and a model that learns nothing meaningful - silently. The fix is discipline: always build your vocab from the same token stream you're encoding, and sanity-check the output with Vocab Decode (the reverse node) once in a while. Round-tripping Encode → Decode and seeing your tokens come back is the single best habit in this pack. Also note the token format is comma-separated, so don't feed it whitespace-separated text and expect clean results - Tokenize emits the right format, so stay in the chain. Niche pack, zero community signal, but this is a dict lookup; the bugs are all upstream in your vocab, not here.