VideoLaVITUnderstandingImage

Feed this node an image, give it a question, and it returns an answer as a plain string. It's the image side of Video-LaVIT's understanding mode: a multimodal LLM that took the same token vocabulary it uses to generate images and learned to read them back. The default prompt - "What is unusual about this image?" - is a decent hint at what it was fine-tuned to do: image Q&A, the classic BLIP-style task.

How it works

The node takes your IMAGE tensor, converts the first frame to a PIL image, and pushes it through the model's image processor. Then it calls the model with {"image": ..., "text_input": prompt} and generates an answer with greedy decoding (num_beams=1, nucleus sampling off). max_length (default 512) caps the answer length, length_penalty (default 1) adjusts how much the model is penalized for producing tokens, and temperature (default 1.0) scales sampling - though with greedy decoding it mostly matters as a knob for determinism between seeds.

Because it returns a STRING output rather than an image, it's not a "generation" node at all - it's an analysis node. The string can go anywhere: a text display node, a condition-into-prompt chain, even a branching decision in an auto-workflow. Want to caption your outputs for archiving? This is the node.

The inputs

• model - a VideoLaVITUnderstanding object from VideoLaVITUnderstandingLoader. Different loader, don't mix it up with the generation one.
• image - any IMAGE input.
• prompt - the question. This is where you get value: ask specific things. "What color is the car?" beats "Describe this image."
• max_length (512), length_penalty (1), temperature (1.0), seed (16) - decoding controls.

Output: STRING - the model's answer, one line of text.

Install

Pack install as always: ComfyUI Manager search "ComfyUI-LaVIT", or git clone https://github.com/chaojie/ComfyUI-LaVIT into custom_nodes. You need the model with its language_model_sft subfolder (see the understanding loader article for the download command), and the manual pip install -r VideoLaVIT/requirements.txt because the pack's root requirements file is empty - Manager won't install the heavy pins on its own.

Where people get burned

The most common mistake is feeding this a VideoLaVIT from the generation loader - the type check will reject it, and rightly so, because the generation model isn't instruction-tuned and doesn't have the same processor. Otherwise, expectations: the model answers in short, often terse sentences, it occasionally misreads details (320p training data), and it's an unmaintained 2024 research wrapper. It's a fun captioning experiment and a genuinely different architecture from CLIP-based encoders - just don't bet your workflow on its accuracy.