VideoLaVITLoader

Every generation node in this pack - T2V, I2V, I2I, and their long variants - demands a VideoLaVIT object on their model input. VideoLaVITLoader is the only thing that produces one. If you're here, you're probably trying to run a workflow that someone shared from this pack and ComfyUI is telling you the model node is missing. This is that node.

What you're actually loading

Under the hood this loads Video-LaVIT v1, a research model from early 2024 (paper: arXiv 2402.03161) that treats video as a mix of keyframes and motion tokens, feeds both to an LLM, and reconstructs frames with a diffusion detokenizer. It's a cool architecture - unified image/video/text generation in one model - but it's a research artifact, not a polished consumer model. It was trained on WebVid-10M: 320p, watermarked, and the model happily reproduces both of those. There's a reason almost nobody talks about it on Reddit; it vanished from the community's attention almost immediately.

The loader builds the generation model with build_model(..., understanding=False), runs it in fp16, and moves the LLaMA backbone, the visual tokenizer, and the tokenizer decoder onto GPU 0. It loads with local_files_only=True, which means it will not auto-download the weights - you have to fetch them yourself.

Getting the weights

The loader reads a folder name from ComfyUI/models/diffusers. The default is Video-LaVIT-v1, and the README's download command is:

huggingface-cli download --resume-download rain1011/Video-LaVIT-v1 \
  --local-dir ~/ComfyUI/models/diffusers/Video-LaVIT-v1 \
  --local-dir-use-symlinks False

If your ComfyUI models dir isn't ~/ComfyUI, point --local-dir at wherever models/diffusers actually lives. This is a multi-GB download - the model bundle includes the generation weights, the SFT understanding weights in a language_model_sft subfolder, and the video_3d_unet.bin detokenizer.

What matters

• model_path - the only input. It's a dropdown populated by listing ComfyUI/models/diffusers, so the folder name you download must match the name you pick. Default is Video-LaVIT-v1.

Output is the custom VideoLaVIT type. Wire it into VideoLaVITT2V, VideoLaVITI2V, VideoLaVITT2VLong, VideoLaVITI2VLong, or VideoLaVITI2I. Note this is the generation model - the separate understanding and detokenizer loaders handle their own paths.

Install and the honest warning

Install the pack via ComfyUI Manager (search "ComfyUI-LaVIT") or:

cd ComfyUI/custom_nodes
git clone https://github.com/chaojie/ComfyUI-LaVIT

Here's the trap: the pack's root requirements.txt is empty. ComfyUI Manager sees that and installs nothing. The real dependencies - pinned to April 2024 versions like diffusers==0.25.1, transformers==4.33.2, timm==0.4.12 - live in VideoLaVIT/requirements.txt, and you must install them manually:

cd ComfyUI/custom_nodes/ComfyUI-LaVIT/VideoLaVIT
pip install -r requirements.txt

Those pins are old enough to fight a modern ComfyUI install over diffusers and transformers. A separate venv or a ComfyUI you're willing to break is the realistic setup. The pack is also a single unmaintained commit from April 2024 - no fixes, no updates. This is tinkerer territory. You need a decent GPU with the VRAM to hold an LLM backbone plus tokenizers in fp16, and the author recommends xformers (which isn't supported on V100s). Finally, remember the README's one hard rule: width and height must be multiples of 64.