Lance Understanding

Lance Understanding is the pack's other output node, and it does the opposite of Lance Generation: instead of turning a prompt into pixels, it looks at an image or video and answers a question about it. Wire it with the x2t_image or x2t_video task in Lance Args, feed it a data loader from an image/video prompt node, and it returns a text answer. In ComfyUI terms it's basically a local vision-language model sitting in your graph - no API, no key, everything on your GPU.

That local part is the appeal. The comfyui-lance-aio pack is a port of ByteDance's Lance-3B, a 3B-active Apache 2.0 model that does generation and understanding in one architecture, and understanding is where a small unified model earns its keep. It won't out-describe a big dedicated VLM - the community consensus at release was that specialized captioners outclass it - but it's free, offline, and shares one weight set with your generation tasks, which is a genuinely unusual trick.

What goes in

• data_loader - from Lance Text Image Prompt (for x2t_image) or Lance Text Video Prompt (for x2t_video). Your question rides in here.
• lance, qwen2_causal_lm, vit - the assembled model, the language model, and the vision encoder. Understanding tasks require the ViT; there's no optional path, unlike generation. So the ViT Loader is mandatory in an understanding graph.
• tokenizer, new_token_ids - the tokenizer plus the special-token IDs from Lance Configure.
• inference_args - the config from Lance Args. Your task (x2t_image or x2t_video) is read from here, and if it's not an understanding task, the node refuses with "task '...' is not an understanding task."

What comes out

A STRING - the model's answer, decoded from generated tokens. Because it's an output node, you also get a UI text display in the node itself, and you can wire the string into anything that takes text (save-to-file nodes, or even another LLM node if you're chaining).

How it works

The node prepares the batch, loads the language model and ViT patchers onto the GPU, and runs Lance's validation path - either the KV-cache or plain variant, per the use_KVcache flag - with greedy decoding (no sampling), capped at 256 tokens of output. The prompt node has already wrapped your question in "Look at the [image/video] carefully and answer the question," and the model generates a text response token by token. Then it decodes, strips the chat-end token, and returns the text. It also unloads the big language model after the run, keeping VRAM free for whatever's next in your graph.

Installing the pack

cd ComfyUI/custom_nodes
git clone https://github.com/SteveImmanuel/comfyui-lance-aio
cd comfyui-lance-aio
pip install -r requirements.txt

or install comfyui-lance-aio via ComfyUI Manager. Understanding needs the Qwen2.5-VL-ViT/ checkpoint folder plus the Lance_3B/ (image) or Lance_3B_Video/ (video) model folder in ComfyUI/models/lance/. The decord dependency is required for video understanding. Restart after installing.

Where people get burned

The task mismatch again, but this time in reverse: setting task to t2i while wiring an understanding graph. Lance Understanding will error out at run time because the task isn't an understanding task. Also, don't feed this node the plain Lance Text Prompt node - understanding needs the image or video prompt variant, or there's no media to look at. And expect the first run to take a while: it loads the ViT plus the full MoE language model, streaming weights if you're on a small card.