Pt Tokenizer

Text doesn't go into a transformer as words - it goes in as token IDs, integers that map to entries in the model's vocabulary. Pt Tokenizer is the node that does that conversion inside ComfyUI-Pt-Wrapper's text pipeline. Give it a list of strings and a tokenizer, and it hands back the token ID tensor and the attention mask tensor that your transformer classification nodes expect.

Two required inputs, both pack-specific types. encode is a PTCALLABLE - a function object, not data. In practice it comes from the pack's "Hf Tokenizer Encode" node, which wraps a Hugging Face tokenizer into a callable and lets you set padding, truncation, and max length there. text_list is a PYLIST of strings (or a single string). The output is a pair: token_tens (the token IDs) and mask_tens (the attention mask marking real tokens vs. padding).

Mechanically this node is embarrassingly simple - it literally calls encode(text_list) and returns whatever comes back. All the real logic lives in the encode callable: padding strategy, truncation, max length, special tokens. That's a design choice worth internalizing: if your tokenized output has ragged shapes or is missing padding, the fix lives in the tokenizer config node upstream, not here.

The honest framing: this node only makes sense inside the Pt-Wrapper universe. PTCALLABLE and PYLIST are types that no other custom-node ecosystem speaks, so the whole tokenize → dataset → train chain has to stay within this pack. That's fine - it's a closed little world, and if you want to do no-code text classification (IMDB sentiment is the pack's flagship example), it's a complete one.

Install

ComfyUI Manager (search "ComfyUI-Pt-Wrapper"), or:

cd ComfyUI/custom_nodes
git clone https://github.com/HowToSD/ComfyUI-Pt-Wrapper

Restart ComfyUI. This is one of the nodes that actually justifies the heavy requirements - transformers, datasets, peft, accelerate, sentencepiece, gensim, plus the rest - because the tokenizer it wraps is a Hugging Face tokenizer. First launch is slow. No model files to download for the pack, though the underlying transformer models load from Hugging Face when you use them.

Common issues

Two things bite people. First, a missing encode connection - the node is dead on arrival without a tokenizer callable, and the error can be cryptic. Second, shape mismatches downstream: if your token tensors and mask tensors disagree in length with what the training node expects, it's padding/truncation settings in the Hf Tokenizer Encode node that need adjusting, not anything here. This node is a pass-through; it faithfully returns whatever its callable produces.