MiniCPM-V Q&A

The one big, serious node in this pack. MiniCPM-V is OpenBMB's vision-language model - point it at an image or a video, ask a question, and get a natural-language answer. It runs entirely on your machine: no API, no paid key. The only thing you need is a free Hugging Face account.

It's two nodes working together: Load MiniCPM-V Model (class DownloadAndLoadMiniCPMV) downloads and loads the model once, then MiniCPM-V Q&A (MiniCPMVNode) does the actual asking.

Setup: the HF auth step everyone trips on

MiniCPM-V isn't on the public Hugging Face hub - OpenBMB gates it behind a click-through license. The load node will hard-fail until you've:

1. Accepted the license on the model's Hugging Face page
2. Created a token at your HF settings
3. Made it available - either an HF_TOKEN environment variable or huggingface-cli login

The error message tells you all of this the moment it happens, which is a nice touch. Model files cache to ComfyUI/models/LLM/, so the download is a one-time thing.

Two model choices, picked on the load node:

• MiniCPM-V (Full) - best quality, heavier VRAM.
• MiniCPM-V-2_6-int4 (7GB VRAM) - quantized with bitsandbytes, ~7GB VRAM footprint. The sensible default on an 8–12GB card.

Inputs that matter

• minicpmv_model (MINICPMV_MODEL) - plug in the load node's output.
• question (STRING) - the actual prompt. Defaults to "Describe this scene in great detail in English".
• image (IMAGE) - a single image or a batch of frames, straight from the graph.
• path (STRING) - a video file, image file, or folder of images read directly from disk. This is the way to go for video: the node samples frames out of it.
• n_frames (INT, default 4) - how many frames to sample: 0 means all (with an OOM warning baked into the tooltip), 1 is the middle frame, 2 is first/last, 3 is first/mid/last, 4+ are evenly spaced. The code caps at 32 and snaps multi-frame counts to a power of two for model compatibility.
• max_tokens (512), temperature (1.0), seed (0 = random) - the standard generation knobs.
• force_offload (default true) - after answering, the model is pushed off the GPU and VRAM is freed. Keep it on unless you're batching and want the model warm.

Output

One STRING - the model's answer. Feed it anywhere text goes: a captioning pipeline, a filename, or just preview it.

Honest caveats

• This is a heavy node. The full model is a multi-gigabyte download, int4 still wants ~7GB of VRAM, and the first generation takes a moment even after loading. You reach for it when a CLIP interrogator's tag list isn't rich enough and you want real sentences about what's in the frame - for auto-captioning, video understanding, or describing a batch of gens.
• trust_remote_code=True. MiniCPM-V needs it; that's normal for this model. But ComfyUI's LLM-vision node scene has a rough security history - the 2024 LLMVISION malware incident - so the sane rule applies: install open-source packs you can actually read. This one is small; give it a skim before you trust it with anything.
• Keep n_frames modest (4–8) unless you enjoy OOM errors, and leave force_offload on. It's the difference between a reusable node and a VRAM hostage.

Install

cd ComfyUI/custom_nodes
git clone https://github.com/drmbt/comfyui-dreambait-nodes

Restart ComfyUI, or search comfyui-dreambait-nodes in ComfyUI Manager. The pack's requirements.txt includes transformers, bitsandbytes, and accelerate - all genuinely needed here - plus a lot of other stuff the grab-bag pulls in. First install is slow; the model download is the second slow thing.

The name is the pitch: a Q&A node for your images and video, entirely local. If you've been captioning with Florence or a tagger and want richer, more human descriptions, this is the upgrade path.