MiniCPM VQA

ComfyUI is very good at making images and very bad at telling you what's in them. MiniCPM-V is the reverse: it's a compact vision-language model from OpenBMB that looks at a picture and talks about it. D_MiniCPM_VQA is the unofficial node that drops that capability into your graph, so "what is this image actually showing?" stops being a trip to a chat website and becomes just another node you can wire things through.

This is one of those nodes people discover sideways. Feed a generated image back in, ask it to describe the scene in detail, and you've got a reverse prompt - a caption you can reuse on the next run. That's the use case the community actually talks about for MiniCPM in ComfyUI. With the 2.6 model it also handles multiple images at once and, fed a batch of video frames, will tell you what's happening in a clip. If you've ever wanted a local model that can do OCR, caption a frame, or compare two renders, this is the one-box answer.

How it works

Under the hood it's a thin wrapper around Hugging Face Transformers' AutoModel/AutoTokenizer. You pick a model from the dropdown - six options from the openbmb Hub repos, spanning the original MiniCPM-V, version 2, the Llama3-backed 2.5, the 2.6, and int4-quantized variants of the last two. On first use the node downloads the checkpoint automatically into ComfyUI/models/prompt_generator/, then loads it with trust_remote_code=True. That flag matters: MiniCPM's Hub files aren't stock transformers, they carry custom modeling code the library executes at load time. The node also caches whatever model you loaded, so switching the dropdown forces a fresh load - and a fresh multi-gigabyte download if you haven't fetched that one yet.

The inputs that actually matter

• image - the obvious one, but there's a catch. For MiniCPM-V, 2, and 2.5 only the first image in the batch is used. For 2.6 the whole batch is fair game, which is exactly how you do multi-image or video.
• text - your question or instruction. Multi-line, so "describe this in the style of a prompt, including lighting and camera angle" works fine.
• model - the six-way dropdown. Default to the 2.6-int4 if you're on a smaller card, the full 2.6 if you've got headroom.
• video_max_num_frames - 0 means "plain image." Set it to a positive number and the node treats the batch as frames, uniformly sampling down to that count before handing them to 2.6. The bundled example uses 64, fed from VHS_LoadVideo through an ImageBatch node.
• temperature - default 0.7, just the sampler temperature for the chat generation.

The output is a single STRING: the model's answer. Wire it into a text preview node, or loop it back into a prompt box for a crude-but-effective reverse-prompting pipeline.

Installing it

ComfyUI Manager covers this one - search for minicpm and install. Or do it by hand:

cd ComfyUI/custom_nodes
git clone https://github.com/hay86/ComfyUI_MiniCPM-V
pip install -r requirements.txt

then restart ComfyUI. One thing worth noting: the requirements file is deliberately thin and unpinned (just torch, transformers, huggingface_hub, and friends), so this pack rides whatever Python environment ComfyUI already has instead of clobbering it - a lot more polite than some nodes, and it dodges a chunk of the dependency hell that plagues the ecosystem. The models auto-download, but they're not small: expect a 10+ GB first fetch for the full 2.6 (the int4 builds are a few GB), and the README asks for 16 GB+ of VRAM on 2.5 and 2.6.

Where people get burned

The first-run "hang" is usually just the model downloading, not a crash - the download lands in ComfyUI/models/prompt_generator/, so if you'd rather use the hf CLI to pre-seed that folder, go for it. On a machine without CUDA it silently falls back to CPU, and an 8B VLM on CPU is a great way to test your patience. And the trust_remote_code flag deserves real respect: it executes code from the model repo. The risk here is low - openbmb is a legitimate, well-known lab - but ComfyUI's history with malicious nodes is real, so don't go swapping the model string for some random clone on the Hub. Finally, the pack has been dormant since mid-2024: it stops at 2.6 and has no idea the newer MiniCPM-o/4.5 exists. That's fine for this use case; just don't expect updates.

For a generalist VQA node inside ComfyUI, this is the one. If your only goal is bulk-captioning a LoRA dataset, though, the community has moved to purpose-built captioners like JoyCaption or Florence 2 - this is the flexible tool you reach for when you want to actually converse with an image, not just tag it.