VLM Prompt (GGUF)

"VLM Prompt (GGUF)" is the headline node of the ComfyUI-VLMPrompt pack: feed it an image or a clip of video, and a vision-language model running entirely on your machine writes a prompt back. No API call, no API key, no cloud round-trip. The "VLM" is short for vision-language model, not an endpoint - the pack name has been confusing people since day one.

If you've ever stared at a frame from a Wan or LTX workflow and tried to describe the motion in a sentence the video model actually respects, you know why this exists. Hand-writing a good I2V prompt is fiddly; letting a small local model do the describing and the embellishing is faster, and it's the reason this pack defaults to the "Wan I2V Prompt" preset. It's a niche pack - don't expect a big community around it - but for the image-to-video prompt loop it does exactly one job well.

How it works

Under the hood it's llama-cpp-python with a multimodal chat handler plus a separate vision projector file (an mmproj GGUF). On the first run the node auto-downloads both the quantized model and its mmproj from Hugging Face into ComfyUI/models/llm/GGUF - that first load can pull a couple of gigabytes, so don't panic when nothing happens for a while. Your image (or evenly sampled frames from the video input) get base64-encoded and sent to the model alongside your chosen preset. On the way out, an output cleaner strips chat-template tokens and leaked <think> blocks, so what lands on the prompt pin is usable text.

The default model is Gemma4-E4B-uncensored-Q4_K_M - a 4B-class quant, small enough for the whole pipeline to stay on one consumer card. You'll notice every bundled model is an uncensored or "abliterated" variant, and there's an explicit "NSFW I2V Tease" preset in the list. That's this pack's lane; choose your own defaults accordingly.

The inputs that matter

• preset_prompt - 27 built-in prompt styles from system_prompts.json: Wan I2V Prompt (default), Describe for Video, 🖼️ Tags, Detailed/Ultra Detailed Description, Enhance, and more.
• custom_prompt - your own instruction. Fills over the preset if you type anything.
• max_tokens - how long the answer can be. 768 is fine for most presets.
• keep_model_loaded - leave this off for one-off runs; turn it on if the node sits in a loop, otherwise you eat a model reload every execution.
• image / video (optional) - wire either in and the model actually looks at it. Text-only runs work too.

The single output, prompt (STRING), goes straight into a Text Encode node or the prompt input of a Wan/Flux workflow.

Installing it

ComfyUI Manager, search "VLMPrompt", install, restart. Or the manual route:

cd ComfyUI/custom_nodes
git clone https://github.com/mexxmillion/ComfyUI-VLMPrompt.git
pip install -r ComfyUI-VLMPrompt/requirements.txt

Here's the gotcha that trips everyone: llama-cpp-python is not in requirements.txt. The README is explicit - the GGUF backend needs a multimodal build that includes the Qwen/Gemma VLM chat handlers, and the stock PyPI wheel may not have them. Install a compatible build yourself or the GGUF nodes will fail on load while the HF, Muse, and VRAM nodes still work fine.

Common issues

• "Images ignored" - the console warns when images arrive but no mmproj is loaded; the model then runs text-only. Fix your llama-cpp-python build, not your workflow.
• Slow first run - that's the multi-GB download, not a hang.
• Opinionated presets - the README says review system_prompts.json before sharing workflows. It's not kidding.