Extensions/ComfyUI-VLMPrompt
ComfyUI Extension

ComfyUI-VLMPrompt

ComfyUI custom nodes for local VLM prompt generation with GGUF and Hugging Face backends.

By mexxmillion·Created 3 months ago·Updated 3 months ago· 2
mexxmillion/ComfyUI-VLMPrompt
Nodes8
On cloudLocal install
CategoryVLMPrompt
Stars2
Updated3 months ago

Nodes (8)

Muse (GGUF)

Muse (GGUF)

VLMPrompt
Muse Advanced (GGUF)

Lighting, eras, palettes, and the full sampler

VLMPrompt
VLM Prompt (GGUF)

The local vision model that turns your frames into prompts — no API, no key

VLMPrompt
VLM Prompt Advanced (GGUF)

The same local VLM prompt node, with the sampling panel turned up

VLMPrompt
VLM Prompt HF (PyTorch)

The transformers-backed VLM prompt node, for when the GGUF handlers fight you

VLMPrompt
VLM Prompt HF Advanced (PyTorch)

HF prompt node, advanced panel — quantization, device, and the sampler in one place

VLMPrompt
VRAM Purge (Inline)

Force ComfyUI to actually give VRAM back

VLMPrompt
VRAM Purge (Start)

Purge once at the start of the workflow, before the VRAM hogs wake up

VLMPrompt
Readme

ComfyUI-VLMPrompt

Custom ComfyUI nodes for generating image and video prompts with local vision-language models.

VLMPrompt can run GGUF multimodal models through llama-cpp-python chat handlers or Hugging Face VLMs through transformers. It also includes Muse prompt-building nodes and VRAM purge helper nodes for longer workflows.

Nodes

  • VLM Prompt (GGUF): simple image/video/text-to-prompt node using GGUF models.
  • VLM Prompt Advanced (GGUF): GGUF node with device and sampling controls.
  • VLM Prompt HF (PyTorch): Hugging Face transformers backend.
  • VLM Prompt HF Advanced (PyTorch): Hugging Face backend with device, quantization, and frame controls.
  • Muse (GGUF): prompt idea generator from genre, mood, artist, era, and subject choices.
  • Muse Advanced (GGUF): full-control Muse prompt generator.
  • VRAM Purge (Inline): passes a value through after clearing CUDA cache.
  • VRAM Purge (Start): workflow-start trigger that clears CUDA cache before downstream nodes run.

Install

Clone this repository into your ComfyUI custom_nodes directory:

cd ComfyUI/custom_nodes
git clone https://github.com/mexxmillion/ComfyUI-VLMPrompt.git

Install the common Python dependencies into the same Python environment ComfyUI uses:

pip install -r ComfyUI-VLMPrompt/requirements.txt

Restart ComfyUI after installing.

GGUF Backend

The GGUF backend expects a multimodal llama-cpp-python build with Qwen/Gemma VLM chat handlers. The normal PyPI package may not include every handler used here.

Install a compatible build, then edit models.json if you want to change the model list, Hugging Face repositories, filenames, model folder, or default generation settings.

By default, GGUF files are downloaded under:

ComfyUI/models/llm/GGUF

Hugging Face Backend

The Hugging Face backend uses transformers, accelerate, huggingface_hub, and PyTorch from your ComfyUI install. Optional 4-bit and 8-bit modes require a working bitsandbytes installation for your platform.

Edit models_hf.json to change the model list. By default, Hugging Face model snapshots are downloaded under:

ComfyUI/models/LLM/HF

If the Hugging Face backend dependencies are unavailable, ComfyUI still loads the GGUF, Muse, and VRAM helper nodes.

Presets

Prompt presets live in system_prompts.json. The browser extension in web/js/vlmprompt.js keeps the selected preset and system prompt widget synchronized for the main GGUF nodes.

Qwen Thinking Tokens

For Qwen text-only runs, the nodes automatically add <|think_off|> and /no_think controls unless you already supplied a thinking toggle in the system or user prompt. The <|think_off|> control is designed to work with fixed Qwen chat templates such as froggeric/Qwen-Fixed-Chat-Templates; /no_think remains as a fallback for runtimes/templates that recognize it.

The output cleaner also strips common leaked thinking artifacts, including <think>...</think>, <thinking>...</thinking>, spaced closing tags, self-closing thinking tags, and orphan closing tags.

Muse prompt data lives in:

  • muse_data.json
  • pose_bank.json

Model Catalogs

Model entries are intentionally data-driven:

  • models.json: GGUF models, mmproj files, chat handlers, stop tokens, and generation defaults.
  • models_hf.json: Hugging Face repositories, thinking-strip behavior, and rough VRAM notes.

You can add your own model entry without changing Python code as long as the backend supports its architecture and chat format.

Notes

  • Large model files are not included in this repository.
  • Generated caches, __pycache__, and local model folders are ignored by git.
  • Some bundled presets are opinionated. Review system_prompts.json before sharing workflows.

License

MIT