VideoLaVITUnderstandingLoader

Video-LaVIT is a unified model - it generates video and it also understands images and video, answering questions about what it sees. VideoLaVITUnderstandingLoader loads the second half of that bargain. It's the sibling of VideoLaVITLoader, but instead of building the generation model it builds the instruction-tuned vision model, and it outputs a different custom type: VideoLaVITUnderstanding. That feeds VideoLaVITUnderstandingImage and VideoLaVITUnderstandingVideo, the two Q&A nodes.

How it works - and why the weights are special

The loader calls build_model(..., understanding=True) and - key detail - points model_path at the language_model_sft subfolder inside the model directory. That's the supervised-fine-tuned checkpoint, released separately from the base weights. The base rain1011/Video-LaVIT-v1 download includes it, but the folder has to exist and be populated or the loader fails at the path join. It also runs in bf16 rather than the generation model's fp16, because the understanding checkpoint is a different, LLaMA-based setup tuned for instruction following.

There's one input beyond the path: max_video_clips (default 16). That's how many 24-frame video chunks the understanding model is allowed to consume when you ask it about a video. If you're asking about a long video, that's the cap on how much of it the model actually sees.

The inputs

• model_path - dropdown populated from ComfyUI/models/diffusers, same as the generation loader. Default Video-LaVIT-v1. The SFT weights live under <model_path>/language_model_sft.
• max_video_clips - video context budget for understanding. Leave at 16 unless you're hitting memory limits or processing very short clips.

Output: the VideoLaVITUnderstanding custom type, wired into the two understanding nodes.

Install

Standard pack install: ComfyUI Manager search "ComfyUI-LaVIT", or git clone https://github.com/chaojie/ComfyUI-LaVIT into custom_nodes. Download the model with:

huggingface-cli download --resume-download rain1011/Video-LaVIT-v1 \
  --local-dir ~/ComfyUI/models/diffusers/Video-LaVIT-v1 \
  --local-dir-use-symlinks False

and confirm the language_model_sft folder landed inside. Then manually pip install -r VideoLaVIT/requirements.txt - the pack's root requirements file is empty, so ComfyUI Manager won't install the pinned deps for you.

Where people get burned

The path is the trap. If you downloaded the model before the SFT release, or a partial download skipped the subfolder, the loader throws on a missing directory and it looks like a dependency problem when it's actually a weights problem. Re-run the full download. Also temper expectations: this is a 2024 research model answering about 320p watermarked training data. It'll caption and answer questions, but it's not GPT-4V. It's here because the same model that generated your video can then tell you what it generated - which is a genuinely neat trick for a research demo, and honestly the most interesting thing this pack does.