APNext MiniCPM Video

This is the video sibling of MiniCPMImageNode: hand it a clip and it tells you what happens across it. MiniCPMVideoNode (APNext MiniCPM Video) samples frames from your video, feeds them to OpenBMB's MiniCPM-V on your GPU, and answers a question about the whole thing. It's a local video-understanding node - useful for auto-captioning clips for a video-model dataset, or writing a prompt that describes motion and progression rather than a single frame.

Video captioning is a different beast from image captioning because you have to summarize time, not just a picture. MiniCPM-V 4.5 has real video understanding built in, and this node exposes the frame-sampling knobs that make it work.

How it works

The node reads your video, samples frames, packs them into groups the model can digest, and asks MiniCPM-V your question. It runs locally on cuda, downloading the weights from HuggingFace on first use.

The inputs that actually matter:

• question (default "Describe the video in detail.") - change it to whatever you want: "Describe the camera movement and how the scene changes."
• fps (default 5) and max_num_frames (default 180) - how densely you sample and the ceiling on total frames. More frames = more temporal detail but more VRAM and time. For a short clip, the defaults are fine; for a long one, either drop fps or you'll hit the frame cap anyway.
• max_num_packing (default 3) - how many frames get packed together per unit the model sees. A performance/detail tradeoff; leave it at the default unless you're tuning.
• enable_thinking - deep-reasoning mode, slower. Off for plain description.
• unload_after_inference - free VRAM when done. Turn it on if the GPU is also running your generation.

You feed the clip through one of the optional inputs - video (a VIDEO or IMAGE-batch connection), video_path, or video_from_input - depending on how your graph sources the footage. precision (bfloat16 / float16) and device behave as they do on the image node.

Two outputs: response (the description STRING) and frame_info (details on which frames were actually sampled - handy for sanity-checking that it saw enough of the clip).

Installing it

ComfyUI Manager: search comfyui_dagthomas, install, restart. Or clone and pip install -r requirements.txt. This node needs decord (video decoding) and transformers from that requirements file, which is exactly why the pack's install list is so heavy - so let it install fully. No API key; the model downloads from HuggingFace on first run.

Common issues

Video won't load / decord error. decord has to be installed and able to read your file. If the manual install skipped it or errored, re-run pip install -r requirements.txt and check for a decord failure in the log. Feeding a container format it can't decode also fails - try a standard MP4.

Out of memory on long clips. More frames means more VRAM. Lower fps and max_num_frames, turn unload_after_inference on, and use bfloat16. A 3-minute clip at 5fps is 900 frames before the cap kicks in - that's a lot to hold at once.

Only saw part of the clip. Check frame_info. If the cap trimmed your video, the model literally didn't see the end. Lower fps so max_num_frames covers the full duration.

First run is slow. Weights downloading from HuggingFace. One-time cost; it's cached after.