Vocab Build

A vocabulary is just a lookup table - but it's the lookup table: the thing that turns words into numbers a model can do math on. Vocab Build is where the pack builds that table. Feed it your tokens, and it counts how often each one appears, keeps everything above a frequency threshold, reserves the special tokens you'll need for padding and sentence boundaries, and emits a cdlVocab object plus its size. It's the d2l Vocab constructor as a node, and everything downstream in the pack's NLP demos depends on it.

The mechanics are worth understanding because they explain three surprises you'll hit. First, <unk> is always added automatically - it's the bucket for words the vocab has never seen, and it means "unknown token" in a way that doesn't crash the pipeline. Second, frequency filtering: set min_freq to 2 and any word appearing only once is dropped to <unk> at encode time - a crude but real way to control vocab size. Third, the reserved tokens (<pad>, <bos>, <eos> by default) get in unconditionally, because you need a padding token for batching even if it never appears in your corpus. Watch how vocab size changes when you tweak min_freq and you've internalized the core NLP tradeoff: coverage vs. size.

Inputs

• tokens_text - a multiline STRING of your token stream. One token per line, or comma-separated tokens per line. The natural source is the output of Tokenize.
• min_freq - minimum frequency to keep a token, default 1 (i.e. keep everything).
• reserved_tokens - comma-separated special tokens, default <pad>,<bos>,<eos>. Add your own if your model needs more.

Outputs

• vocab - the cdlVocab object, which you feed straight into Vocab Encode and Vocab Decode. Don't try to read it as a string; it's an opaque object in this pack's type system.
• vocab_size - an INT, the total number of tokens in the table. This is the number you'll wire into model nodes that need to know the input/output dimension of their embedding layer.

Installing it

Part of ComfyDL. ComfyUI Manager, search "ComfyDL", or:

cd ComfyUI/custom_nodes
git clone https://github.com/Cynthia-lxx/ComfyDL
pip install -r ./ComfyDL/requirements.txt

Restart. Only matplotlib to install, no model files.

Gotchas

The subtlest thing here is that the vocab is sorted alphabetically, not by frequency. The d2l implementation builds idx_to_token as a sorted set, so your most common words aren't index 0, 1, 2 - and vocab_size tells you the count but not the ordering. That's fine for correctness, but it's a surprise if you expect word2vec-style frequency ordering. Also: since everything unknown maps to <unk> at encode time, a vocab built on a tiny corpus will map a lot of your actual text to <unk> - that's not a bug, that's the lesson. One practical workflow note: because the vocab is a custom cdlVocab type, it only flows to other ComfyDL nodes, so keep the whole chain (Build → Encode → model) inside the pack. Young, no community signal, but the node is textbook collections.Counter logic - the failure modes are all in how you prepare your text, not in the node.