ComfyUI-VLMPrompt

Custom ComfyUI nodes for generating image and video prompts with local vision-language models.

VLMPrompt can run GGUF multimodal models through llama-cpp-python chat handlers or Hugging Face VLMs through transformers. It also includes Muse prompt-building nodes and VRAM purge helper nodes for longer workflows.

Nodes

• VLM Prompt (GGUF): simple image/video/text-to-prompt node using GGUF models.
• VLM Prompt Advanced (GGUF): GGUF node with device and sampling controls.
• VLM Prompt HF (PyTorch): Hugging Face transformers backend.
• VLM Prompt HF Advanced (PyTorch): Hugging Face backend with device, quantization, and frame controls.
• Muse (GGUF): prompt idea generator from genre, mood, artist, era, and subject choices.
• Muse Advanced (GGUF): full-control Muse prompt generator.
• VRAM Purge (Inline): passes a value through after clearing CUDA cache.
• VRAM Purge (Start): workflow-start trigger that clears CUDA cache before downstream nodes run.

Install

Clone this repository into your ComfyUI custom_nodes directory:

cd ComfyUI/custom_nodes
git clone https://github.com/mexxmillion/ComfyUI-VLMPrompt.git

Install the common Python dependencies into the same Python environment ComfyUI uses:

pip install -r ComfyUI-VLMPrompt/requirements.txt

Restart ComfyUI after installing.

GGUF Backend

The GGUF backend expects a multimodal llama-cpp-python build with Qwen/Gemma VLM chat handlers. The normal PyPI package may not include every handler used here.

Install a compatible build, then edit models.json if you want to change the model list, Hugging Face repositories, filenames, model folder, or default generation settings.

By default, GGUF files are downloaded under:

ComfyUI/models/llm/GGUF

Hugging Face Backend

The Hugging Face backend uses transformers, accelerate, huggingface_hub, and PyTorch from your ComfyUI install. Optional 4-bit and 8-bit modes require a working bitsandbytes installation for your platform.

Edit models_hf.json to change the model list. By default, Hugging Face model snapshots are downloaded under:

ComfyUI/models/LLM/HF

If the Hugging Face backend dependencies are unavailable, ComfyUI still loads the GGUF, Muse, and VRAM helper nodes.

Presets

Prompt presets live in system_prompts.json. The browser extension in web/js/vlmprompt.js keeps the selected preset and system prompt widget synchronized for the main GGUF nodes.

Qwen Thinking Tokens

For Qwen text-only runs, the nodes automatically add <|think_off|> and /no_think controls unless you already supplied a thinking toggle in the system or user prompt. The <|think_off|> control is designed to work with fixed Qwen chat templates such as froggeric/Qwen-Fixed-Chat-Templates; /no_think remains as a fallback for runtimes/templates that recognize it.

The output cleaner also strips common leaked thinking artifacts, including <think>...</think>, <thinking>...</thinking>, spaced closing tags, self-closing thinking tags, and orphan closing tags.

Muse prompt data lives in:

• muse_data.json
• pose_bank.json

Model Catalogs

Model entries are intentionally data-driven:

• models.json: GGUF models, mmproj files, chat handlers, stop tokens, and generation defaults.
• models_hf.json: Hugging Face repositories, thinking-strip behavior, and rough VRAM notes.

You can add your own model entry without changing Python code as long as the backend supports its architecture and chat format.

Notes

• Large model files are not included in this repository.
• Generated caches, __pycache__, and local model folders are ignored by git.
• Some bundled presets are opinionated. Review system_prompts.json before sharing workflows.

License

MIT