MiniCPM VQA Polished

This node drops a real vision-language model into your graph: hand it an image, a video, or just plain text, ask it a question, and it writes back an answer. No API key, no Ollama server babysat in a second terminal, no copy-pasting a description back into a prompt box by hand. That last part is the reason people actually reach for this: running a reference image through a VLM to get a written description, then wiring that text straight into your next CLIPTextEncode, is a real workflow - it's the same move people make with "minicpm-v or qwen2 vl" to auto-write a video-appropriate prompt from a source image before handing it to something like LTX.

Under the hood it's MiniCPM-V-4.5, OpenBMB's vision-language model, wrapped so it eats ComfyUI's native IMAGE/VIDEO tensors instead of file paths. Mechanically it's the same shape as any multimodal chat model: the image gets encoded into visual tokens, those get mixed in with your text tokens, and an autoregressive language model decodes an answer out of that combined context. The wrapper's whole job is loading the checkpoint and doing that tensor plumbing so you never see any of it.

The inputs that matter

• text - your question, multiline. It's a chat model, not a fixed captioner, so phrase it like you're actually asking something: "Describe this image" works, "what's the lighting like and is it indoors or out?" also works.
• model - MiniCPM-V-4_5-int4 (default) or the full MiniCPM-V-4_5. int4 is the quantized build: smaller download, lower VRAM, faster. For "what's in this image" style questions the quality gap versus full precision is small enough that int4 is the sane default; reach for the full model when you need finer judgment calls and have the VRAM to spare.
• source_image / source_video (optional) - plug in one. This is the whole point of "Polished" versus the pack's older MiniCPM_VQA node: instead of three separate numbered image sockets, you get one source_image slot. Need more than one image? Batch them first with this pack's own MultipleImagesInput node and feed the batch in here.

The rest of the required fields are standard LLM sampling knobs, not things you'll touch often: top_p, top_k, temperature (0.7 default - drop it toward 0 for flatter, more literal answers), repetition_penalty, and max_new_tokens as your leash on rambling. video_max_num_frames / video_max_slice_nums only matter with source_video - they cap how many frames get sampled rather than every frame, which would otherwise eat your VRAM on anything longer than a few seconds. keep_model_loaded defaults off, so the model unloads from VRAM after every run - fine once, wasteful ten calls in a row, so flip it on for repeated runs. seed was added specifically so you can reproduce an answer.

Output: one STRING, the model's answer. Wire it into this pack's DisplayText node if you just want to read it, or straight into a CLIPTextEncode if the answer is about to become your next prompt.

Installing it

Via ComfyUI Manager: search minicpm, install, restart. Manually:

cd ComfyUI/custom_nodes
git clone https://github.com/IuvenisSapiens/ComfyUI_MiniCPM-V-4_5
pip install -r ComfyUI_MiniCPM-V-4_5/requirements.txt

then restart. No model files ship with the pack - whichever model you pick downloads automatically on first run, straight into ComfyUI/models/prompt_generator/. Budget time for that: it's a vision-language model, the download isn't small, and a slow first run isn't a crash, it's the checkpoint arriving.

Common issues & troubleshooting

First run looks frozen. It's downloading weights, not hung - check the console for progress before you kill it.

Flaky on a fresh pod every session. A Reddit user tried this pack's earlier version (ComfyUI_MiniCPM-V-2_6-int4, same author) for video captioning and reported it "not working on runpod." The likely culprit isn't the model itself - it's pip installs plus multi-gigabyte re-downloads on a container that resets between sessions. On ephemeral storage, point models/prompt_generator/ at a persistent volume, or accept the download cost every boot.

pip install throws a dependency error. requirements.txt installs into the same Python environment as every other custom node - a version clash (transformers is the usual suspect across the ecosystem) is a generic ComfyUI custom-node problem, not something unique to this pack. Check what version another LLM or vision node in your setup already pinned before assuming this one is broken.

It errors with neither image nor video connected. Text-only queries work fine, but if your prompt assumes visual context that isn't there, you'll get a confused answer rather than an error - rephrase as a pure text question if you're not actually feeding it anything to look at.