Tokenize

Every NLP pipeline in this pack starts the same way: raw text has to become tokens before a vocabulary or a model will touch it. Tokenize is that step. You give it plain text, one sentence per line, and it splits each line into word tokens or character tokens, then hands you back a string with tokens comma-separated per line and lines newline-separated. It's the d2l tokenize function as a node, and it's deliberately basic - no subword BPE, no special tokens, just honest whitespace and character splitting.

That simplicity is the point. The d2l philosophy is that you should build the tokenizer, then the vocab, then the model, and understand each layer before it becomes magic. This node is the layer that makes you confront the fact that "token" is a choice: word mode splits "the quick brown fox" into four whitespace-delimited tokens, char mode splits it into fourteen characters. Watch what that choice does to vocab size downstream and you've learned something real about why modern tokenizers exist.

Inputs

• text - a multiline STRING box, defaulting to a couple of sentences. Each non-empty line is treated as one sentence. So "the quick brown fox\njumps over the lazy dog" becomes two lines of output.
• token_mode - dropdown: word or char. Word splits on whitespace; char splits into individual characters, spaces and all.

Output

A single tokens_str STRING output: one line per input line, tokens joined with commas. You'd normally feed this straight into Vocab Build (CdlVocabBuild), which counts frequencies and builds the index. If your sentences are different lengths - and they will be - the pack's Truncate/Pad node is the next step before batching, because tensors need fixed lengths.

Installing it

It ships inside ComfyDL. ComfyUI Manager, search "ComfyDL", install, restart. Or by hand:

cd ComfyUI/custom_nodes
git clone https://github.com/Cynthia-lxx/ComfyDL
pip install -r ./ComfyDL/requirements.txt

Only matplotlib as a dependency, no model files. Painless.

Gotchas

The comma-and-newline serialization is the thing to get straight: this node hands you a string encoding of your tokens, not a tensor. ComfyDL's vocab nodes (Vocab Encode, Vocab Build) are built to accept that format, so stay within the pack and it flows. If you're expecting a Python list of lists, you'll be confused - read the output as the wire format it is. Also note punctuation isn't stripped: "fox." and "fox" are different tokens in word mode, which bloats the vocab. For a learning exercise that's actually a useful lesson - you'll see your vocab size jump the moment punctuation enters your corpus. This is a young, tiny-community pack, but this node is pure string processing with no failure modes worth fearing; the worst case is a vocab that's bigger than you wanted because you didn't clean your text.