Sp Encode

Feeding text to a model is a two-step job: turn words into token IDs, then decide how to pad and truncate them so a batch is one clean tensor. Sp Encode does both, and it does the ComfyUI-Pt-Wrapper thing of packaging the whole job as a callable you can hand to other nodes.

This is the SentencePiece route through the pack. ComfyUI-Pt-Wrapper (HowToSD's no-code PyTorch training pack) supports two tokenizer flavors - Hugging Face and SentencePiece - and this node is the SentencePiece one. SentencePiece is the tokenizer behind a lot of older and non-Transformer NLP: it doesn't care about your language or even your spaces, it just learns a vocabulary of subword units from raw text. Where Hf Tokenizer Encode lets you name any model on the Hub, Sp Encode needs a trained SentencePiece model file, which you load into the graph with the pack's Sp Load Model node first (that's where spmodel comes from).

How it works. You configure the encode behavior once, here in the graph, and the node emits a PTCALLABLE - a function you then invoke later (typically via Pt Apply Function) with an actual sentence or list of sentences, which returns token-ID tensors and an attention mask. It's a neat trick: the settings are captured at graph-build time, the work happens when the training loop actually needs the tokens.

The inputs that matter:

• spmodel - the SPMODEL from Sp Load Model. Non-negotiable.
• padding / padding_method - pad to equal length (longest pads to the longest sequence in the batch; max_length pads everything to a fixed length).
• padding_value - the token ID used as the pad symbol (0 by default).
• truncation / max_length - cut sequences longer than max_length (512 default). If you pad and truncate, you must set max_length; if you truncate without padding you can still set it.

Output is the PTCALLABLE. When you call it: with padding on, you get a 2D token tensor and a 2D attention mask (1 = real token, 0 = padding). With padding off, a single string gives you 1D tensors and a list gives you a list of them.

Where people get burned. The #1 mistake is forgetting the spmodel dependency: there's no model file shipped with the pack, so Sp Load Model errors until you put a real spiece.model in the pack's models/sentencepiece/ folder (training one is a separate job - the pack's docs walk through it). Second: mixing padding off with padding_method/max_length expectations - if padding is off, no mask is length-normalized, and downstream nodes that assume rectangular batches will throw shape errors. Third: the mask is your responsibility - if you skip padding, feed the returned mask into whatever node computes loss, or padding tokens get counted as signal.

Install: ComfyUI Manager → "ComfyUI-Pt-Wrapper", or:

cd ComfyUI/custom_nodes
git clone https://github.com/HowToSD/ComfyUI-Pt-Wrapper

then restart. sentencepiece is in the pack's requirements, so the install handles it; first install is slow because the stack is heavy (transformers, datasets, sklearn, pinned gensim).

Troubleshooting: "SentencePieceProcessor not found / failed to load model" - file missing or wrong name in models/sentencepiece/. Token tensors with wrong shapes downstream - check whether padding is actually on. If you see odd token IDs like 0 flooding your sequences, you're using padding_value 0 (default) which is also a real token in some vocabularies - pick a value that isn't a real token.