Hf Tokenizer Encode

Want to feed text into a Transformer model inside ComfyUI? Hf Tokenizer Encode is the easiest door in: type a Hugging Face model name, and it produces a tokenizer callable that turns sentences into input_ids and attention masks. No local model file, no training a tokenizer, no separate load step. If you've ever used AutoTokenizer, this is that, as a node.

It's one of two tokenizer nodes in HowToSD's ComfyUI-Pt-Wrapper (the 200-node no-code PyTorch training pack, spin-off of ComfyUI-Data-Analysis). Sp Encode handles SentencePiece; this one talks to the Hugging Face Hub. It's the node the README's from-scratch Transformer example leans on for its IMDB text classification workflow - you grab the tokenizer for a model like bert-base-uncased or distilbert-base-uncased and wire the resulting callable into your model and training nodes.

How it works. You configure encoding behavior in the graph; the node builds a callable (a PTCALLABLE) that accepts a sentence or list of sentences and returns a dict with input_ids and attention_mask (1 = real, 0 = padding). You invoke that callable elsewhere in the graph - Pt Apply Function is the usual spot - so the tokenization happens when the training loop actually needs it.

The inputs that matter:

• model_name - the one you actually set. A Hugging Face repo id like bert-base-uncased, or a local path to a saved tokenizer. First use downloads the tokenizer from the Hub, so you need network that first time.
• padding (default true, unlike the SentencePiece node) / padding_method - pad to max_length or to the longest sequence in the batch.
• truncation (default true) / max_length (default 512). Nice detail: set max_length to 0 and it uses the model's own maximum input length instead of your hard number.

Output is the PTCALLABLE, which pairs with Pt Apply Function and the pack's embedding/model nodes.

The trap most people hit: model_name must match the model you actually use. The README's 85%-accuracy IMDB example is built around a specific tokenizer+model pairing, and mismatched vocabularies produce garbage embeddings that train to random accuracy. Also remember the first run is a network download - if the node hangs or throws a connection error, that's the Hub download, not your workflow. And max_length means different things to different models: a model with a 512-token context doesn't care how big your padding limit is, it just truncates harder. Set max_length to match your model's real window.

Install: ComfyUI Manager → search "ComfyUI-Pt-Wrapper", or:

cd ComfyUI/custom_nodes
git clone https://github.com/HowToSD/ComfyUI-Pt-Wrapper

then restart. The pack installs transformers and datasets in its requirements, so the tokenizer machinery is there; first install is slow and you'll want the pinned gensim noted in case it fights another pack.

Troubleshooting: tokenizer downloads fail - check network/HF access, run it twice (Hub hiccups). Shape errors downstream - verify padding is on if a node expects rectangular batches. Odd or all-padding output - wrong model_name for your model. max_length of 0 producing weird lengths - that's the "use model max" mode working as intended, so don't fight it, read the model card for the real window.