VideoLaVITT2V

This is the headline node of the pack: give Video-LaVIT a sentence and get a short video back. It's the direct port of the model's generate_video demo into ComfyUI. Expect a research-grade result - short, low-res, occasionally watermarked - but it's a real, working text-to-video path on your own GPU, which was rare enough in April 2024 to justify this whole pack existing.

How it works

Video-LaVIT doesn't generate video the way a diffusion model does. It generates tokens: first the LLM writes discrete visual tokens for a keyframe image (at keyframe_width × keyframe_height, default 1024×576), then it writes motion tokens describing how that frame moves. A diffusion-based video detokenizer turns the keyframe plus motion tokens into actual frames at video_width × video_height (default 576×320). Two separate guidance knobs exist because two separate generation steps exist: guidance_scale_for_llm steers token generation, guidance_scale_for_decoder steers the detokenizer's diffusion pass. It's a two-stage pipeline wearing a single-node disguise.

You get roughly a two-second clip - 24 frames per generation - which is why the long variants exist.

The inputs that matter

• prompt - the thing being animated. Default is "FPV drone footage of an ancient city in autumn," and honestly that default is a decent stress test because fast camera motion is where this model shows its weaknesses.
• keyframe_width / keyframe_height - the keyframe resolution. Keep at 1024×576; this is where the "multiple of 64" rule from the README bites hardest.
• video_width / video_height - the output video resolution. 576×320 is the WebVid-native size, so it's the safe zone.
• guidance_scale_for_llm (4) and guidance_scale_for_decoder (7) - the two-stage guidance. Raise the decoder one if the output looks mushy, but don't expect miracles from a model trained on 320p watermarked clips.
• num_inference_steps (50), top_k (50), seed (16) - the usual knobs. seed is genuinely seeded (torch.manual_seed), so you can retry the same prompt with a different seed and get a different take.

The output is IMAGE - a batch of frames, not a file. In the bundled wf_t2v.json, that goes straight into VHS_VideoCombine to be assembled into an actual video.

Install

Same pack install as every other node here - ComfyUI Manager search "ComfyUI-LaVIT", or git clone https://github.com/chaojie/ComfyUI-LaVIT into custom_nodes. The heavy lifting: download the model into models/diffusers/Video-LaVIT-v1 (see the loader article for the exact huggingface-cli command), and manually pip install -r VideoLaVIT/requirements.txt because the pack's root requirements.txt is empty. And you're loading an LLM backbone in fp16, so bring VRAM - xformers is recommended and V100s are out.

Where people get burned

Mostly expectations. This is a single-commit wrapper around an unmaintained research model from a forgotten 2024 wave - it will not compete with modern video models. The specific gripes: watermarks show up because the training data had them; long prompts drift because the LLM is writing token sequences; and if you set dimensions that aren't multiples of 64, the tokenizer will happily reject them. If you load the bundled workflow and it errors on a missing node, the pack's empty requirements.txt is the usual culprit - ComfyUI Manager never installed the pins, so the import fails. Install the subfolder requirements manually and restart.