VideoLaVITI2V

Image-to-video is arguably where this model shines, because it's where the architecture's "everything is tokens" bet pays off. Give VideoLaVITI2V a picture and a sentence and it returns a short clip of that picture moving. It's the node behind the wf_i2v.json workflow, and it's the friendlier of the pack's generation nodes - you're not trusting the model to invent a world, just to move the one you handed it.

How it works

Video-LaVIT represents both images and text as discrete tokens, so a multimodal prompt is just a token sandwich: text tokens + image tokens. VideoLaVITI2V calls multimodal_video_generate with [(prompt, 'text'), (image, 'image')]. The LLM only has to generate the motion tokens - the keyframe is already given to you, unlike T2V where it must also hallucinate the keyframe from nothing. Then the video detokenizer diffuses those motion tokens into 24 frames at video_width × video_height. Because the reference frame is real, the result tends to be more stable than the pure T2V path.

The image input is a standard IMAGE tensor, so anything that produces an image feeds in - LoadImage, another generation node, an upscaler output. The node takes image[0], converts it to a PIL RGB image internally, and hands it to the model.

The inputs that matter

• image - your starting frame. The model is happiest when this matches the video resolution you ask for; a heavily letterboxed or off-aspect source gives you weird crops.
• prompt - the motion instruction. "FPV drone footage of an ancient city in autumn" is the default, and yes, it's really that drone-heavy. You're telling the model how the camera should move, so verbs matter more than nouns.
• video_width / video_height - output size, default 576×320 (the model's native WebVid resolution). Multiples of 64, per the README.
• guidance_scale_for_llm (4) - controls how hard the LLM sticks to the prompt while writing motion tokens. There's no decoder guidance input here (the I2V path runs the detokenizer on the reference frame), so don't go hunting for one.
• num_inference_steps (50), top_k (50), seed (16) - standard sampling knobs.

Output is IMAGE - 24 frames as a batch - which you'll wire into VHS_VideoCombine to get a file. The bundled wf_i2v.json does exactly that: LoadImage → VideoLaVITI2V → VHS_VideoCombine, fed by VideoLaVITLoader.

Install

Pack-level install: ComfyUI Manager, search "ComfyUI-LaVIT", or git clone https://github.com/chaojie/ComfyUI-LaVIT into custom_nodes. Model goes in models/diffusers/Video-LaVIT-v1 (see the loader article for the exact download command). And the recurring trap: the root requirements.txt is empty, so install VideoLaVIT/requirements.txt manually - ComfyUI Manager won't do it for you, and the whole pack fails to import without those pinned deps.

Where people get burned

The image-video quality gap is the big one: the still is yours, but the motion is WebVid-trained, so expect warping, watermarks, and odd physics on complex scenes. Keep the source image close to 576×320 if you can. And set expectations around the pack itself - single commit from April 2024, zero updates since, essentially no community around it. If you want state-of-the-art I2V, this is not it; if you want to poke at a genuinely different architecture that treats images and video as the same token language, it's a fun afternoon.