Lance Tokenizer Loader

Lance Tokenizer Loader is the pack's least interesting node and one of its most load-bearing. It takes a checkpoint directory, loads the Qwen2 tokenizer that lives inside it, and hands you a TOKENIZER. That's the whole job. Every Lance workflow - generation, editing, understanding - needs a tokenizer, because your prompt has to become token IDs before the model can see it, and Lance's vocabulary is not ComfyUI's built-in CLIP vocabulary.

Lance is ByteDance's 3B-active Apache 2.0 model for unified image/video generation, editing, and understanding. The pack, comfyui-lance-aio, is an unofficial ComfyUI port of the official codebase, and the tokenizer is a faithful piece of that: a Qwen2 tokenizer loaded from the checkpoint folder itself. That's the part to internalize - the tokenizer is bundled with the model files, not with ComfyUI.

The one input

• ckpt_dir - a string path to a Lance checkpoint folder. In the example workflows this is wired from the LANCE_CKPT_DIR output of Lance Args, so it automatically points at the right Lance_3B/ or Lance_3B_Video/ folder for your task. Either works, because both checkpoint folders ship their own tokenizer files (tokenizer.json, vocab.json, merges.txt).

The tokenizer doesn't get a model_args or config input, unlike the model loaders. It reads its config straight from the folder. That's why the folder layout from the README is non-negotiable: if tokenizer.json is missing from where you put the checkpoint, this node fails at load time.

What comes out

One TOKENIZER output. It feeds Lance Configure (which uses it to add Lance's special tokens) and, via Lance Configure's NEW_TOKEN_IDS output, every prompt node. You won't connect it directly to the prompt nodes - the new-token-ids side comes out of Configure, not here.

How it works

Under the hood it's a single call: Qwen2Tokenizer.from_pretrained(ckpt_dir). The pack vendors its own Qwen2 tokenizer implementation under modeling/ rather than leaning on ComfyUI's built-in text encoders, because the official Lance codebase expects a specific tokenizer class with specific special-token handling. It loads lazily and cheaply - a tokenizer is tiny compared to the model weights, so this node is fast even on a cold start.

Installing the pack

This node only exists as part of comfyui-lance-aio:

cd ComfyUI/custom_nodes
git clone https://github.com/SteveImmanuel/comfyui-lance-aio
cd comfyui-lance-aio
pip install -r requirements.txt

ComfyUI Manager can install it too, as comfyui-lance-aio. After cloning, download the Lance checkpoints from bytedance-research/Lance into ComfyUI/models/lance/ and restart ComfyUI. The requirements file pins transformers>=4.50.3,<5 - keep it that way; the vendored tokenizer code is tied to a transformers version under 5.

Troubleshooting

The failure mode here is a missing file: "No such file" pointing at a tokenizer file inside the checkpoint folder. That's a layout problem, not a code problem - check that ckpt_dir points at the inner Lance_3B folder, not .../lance, and that the folder wasn't stripped down (some people try to save disk by deleting the tokenizer files because they look redundant. They're not). If the node loads but downstream generation produces garbage or token-ID errors, double-check that Lance Configure is running between the loader and the prompt nodes - the tokenizer has to have Lance's special tokens added before it's used, and that's Configure's job.