VideoLaVITUnderstandingVideo

Image Q&A nodes are everywhere. Video Q&A nodes are not. VideoLaVITUnderstandingVideo takes a path to a video file on disk and a question, and answers with what's happening in the clip. It's the video half of Video-LaVIT's understanding mode, and it's the node that makes the pack feel genuinely unusual - the same model family that generates video also reads it back. It even comes with a path-helper sibling (VHS_FILENAMES_STRING_LaVIT) to feed it a path from Video Helper Suite.

How it works

Unlike the image node, which takes a tensor, this one takes a file path - video_path is a plain string, which is why the VHS adapter node exists. The model internally processes the video with its video processor, chunking it into the clip count configured on the understanding loader (max_video_clips), and generates an answer the same way as the image node: greedy decoding, max_length cap, length_penalty, temperature. Output is a STRING.

The default prompt - "What is the man doing in this video?" - is a template from the model's own demos. It works, but you'll get better answers with targeted questions, same as any vision model.

The inputs

• model - a VideoLaVITUnderstanding from VideoLaVITUnderstandingLoader. The loader's max_video_clips setting determines how much of the video gets looked at, so if answers are wrong on long clips, that's your first lever.
• video_path - absolute path to a video file on disk. Not an image, not a batch of frames. Multiline field, but one path is what it wants.
• prompt - the question.
• max_length (512), length_penalty (1), temperature (1.0), seed (16).

Output: STRING.

Install

Pack install is the usual: ComfyUI Manager search "ComfyUI-LaVIT", or git clone https://github.com/chaojie/ComfyUI-LaVIT into custom_nodes. Model needs the language_model_sft subfolder (see the understanding loader article). And the recurring install trap: the pack's root requirements.txt is empty, so you must pip install -r VideoLaVIT/requirements.txt yourself or nothing in the pack imports.

Where people get burned

The path handling is the classic stumble. Give it a relative path and it may resolve from the wrong working directory; use an absolute path. And if you're feeding it a video that's very long, the model only sees as many clips as max_video_clips allows - the answer might describe the first few seconds and nothing after. Otherwise, set expectations like you would for any 2024 research vision model: short answers, occasional misreads, unmaintained wrapper. But as a demo of "one token vocabulary reads and writes video," it's one of the most interesting things in this entire pack.