ViT Loader

ViT Loader is how Lance sees. It loads the Qwen2.5-VL vision transformer - the "ViT" in the name - that turns images and video frames into tokens the language model can reason over. If your Lance task involves looking at anything, this node is on the required path: understanding, image-to-video, and every edit task all need it. Only pure text-to-image and text-to-video can skip it.

The comfyui-lance-aio pack ports ByteDance's Lance-3B, a 3B-active Apache 2.0 model that fuses a language model with a vision encoder. The ViT is one of the three weight blocks that make up the "9B total" people argue about - roughly 3B for the vision encoder, 3B for image generation, 3B for video. It's the same architecture family Qwen2.5-VL uses, which is why the pack can load it with ComfyUI's own bundled Qwen2VLVisionTransformer rather than vendoring yet another copy.

The one input

• ckpt_dir - path to the Qwen2.5-VL-ViT checkpoint folder (wire it from Lance Args' VIT_CKPT_DIR). The loader reads two files from it: vit.safetensors for the weights and config.json for the architecture settings.

What comes out

Two outputs, both feeding the Lance Loader:

• VIT - the loaded vision transformer, wrapped in its own model patcher with bfloat16 compute and offload support.
• VIT_CONFIG - the model config read from the folder. The Lance Loader needs it to construct the full Lance model, so wire both outputs even though it feels redundant.

How it works

The loader builds the vision transformer on the offload device in bfloat16, patches it through comfy.ops.manual_cast so weights can stream, and loads vit.safetensors. There's one compatibility shim in here worth knowing about: the pack subclasses ComfyUI's vision transformer to adapt a function-signature mismatch between ComfyUI's implementation and what Lance's code calls. That's a maintenance detail that means this node is a bit more fragile to ComfyUI core updates than the rest of the pack - if a ComfyUI update ever breaks the Lance nodes, this is a prime suspect.

For understanding tasks the ViT is strictly required (the Lance Understanding node has no optional path). For generation tasks that take an image input - image-to-video and edits - it's also needed, because the model has to encode your reference. The Lance Generation node lists vit as optional, but "optional" only means optional for text-only tasks.

Installing the pack

cd ComfyUI/custom_nodes
git clone https://github.com/SteveImmanuel/comfyui-lance-aio
cd comfyui-lance-aio
pip install -r requirements.txt

or comfyui-lance-aio via ComfyUI Manager. Then download the Lance checkpoints from bytedance-research/Lance into ComfyUI/models/lance/, making sure Qwen2.5-VL-ViT/ contains both config.json and vit.safetensors. The pack pins transformers>=4.50.3,<5, and since this node constructs a config from a transformers class, that pin is load-bearing here. Restart ComfyUI and the nodes appear under the Lance category.

Troubleshooting

If this node errors on a missing config or a shape mismatch, the folder is usually incomplete - a half-downloaded vit.safetensors or a missing config.json. Redownload the folder rather than copying files from other repos; the config has to match these weights. And if the node fails right after a ComfyUI update, check the GitHub issues - the compatibility shim is the kind of thing that breaks upstream and gets patched downstream.