Unified Vision Prompt Generator

The name is a lie in the best way. "Unified Vision Prompt Generator" sounds like a SaaS widget, but this node never calls an API and needs no key. It's one box that loads a real vision-language model from HuggingFace, points it at your image, and hands you back a natural-language description as a prompt STRING you can wire anywhere.

You reach for it for one of two jobs. First: turn an image into the prompt that produced it, then feed that straight into an img2img pass, an outpaint, or an editor like Qwen-Image-Edit. Second - and this is where it earns its keep - captioning a dataset for LoRA training. The newer LLM-encoder models (Flux, Z-Image, Krea 2) read sentences, not booru tags. A model that writes like a person gives you captions the encoder actually understands, which is exactly the "natural language, not tags" advice that keeps showing up in training guides. If your dataset is photos or real-world subjects, this beats a tagger.

How it works

The model dropdown lists ten weights across three families: the Qwen2-VL / Qwen2.5-VL line (2B up to 72B), Moondream 1 and 2, and SmolVLM (256M to full Instruct). Pick one, feed an image into the optional IMAGE input, and the node auto-downloads the weights from HuggingFace on first use, runs the VLM on ComfyUI's current GPU device, and returns a single prompt string. No image? Qwen and SmolVLM will still answer your question from text alone; Moondream just refuses and tells you it needs an image.

The inputs that actually matter

• model - defaults to Qwen/Qwen2.5-VL-7B-Instruct, which is a lot of model for the average card. See below.
• question - the instruction, default "Describe this image in detail." This is where you steer the output: "describe the lighting and camera angle", "write a caption for training, no brand names", etc.
• max_new_tokens (default 1024) and temperature (default 0.7) - the usual generation knobs. Lower temperature for factual captioning, raise it a bit if you want more creative prose.
• keep_model_loaded - off by default, and the thing to flip on if you're batching. Off means the model is shoved back to CPU and cleared after every single call.
• seed - it gets SHA-256 hashed before seeding torch, so reproducibility is "same seed, roughly same output", not bit-exact across versions.

Install

Standard two routes: ComfyUI Manager (search "Unified Vision Prompt Generator"), or the manual clone:

cd ComfyUI/custom_nodes
git clone https://github.com/GraftingRayman/Unified-Vision-Prompt-Generator

Restart ComfyUI. Here's the gotcha: the pack ships no requirements.txt, so Manager won't install its dependencies for you. transformers is usually already in ComfyUI's environment, but the Qwen models need qwen-vl-utils, and the code will raise a clear error telling you to install it if it's missing:

pip install qwen-vl-utils

And remember every model auto-downloads to your HuggingFace cache on first use - several GB for the 7B, no progress bar in ComfyUI. It looks frozen; it isn't.

Where people get burned

• The default model is heavy. 7B Qwen on an 8–12 GB card is a tight squeeze. The author develops on 16 GB and explicitly says they couldn't test the 72B. If VRAM is your problem, drop to Qwen2.5-VL-3B-Instruct or a SmolVLM variant - they're dramatically faster and still write coherent captions.
• It re-runs every queue pass. IS_CHANGED returns NaN, so the node never caches its result. With keep_model_loaded off, that's unload → reload → download-from-cache churn on every execution. If you're chaining several calls, flip keep_model_loaded on - just know it pins VRAM until you clear it.
• Dependency hell is real here. Every custom node shares one Python environment, and a version-pinned transformers from another pack can silently break this one. If it starts erroring after you install something else, that's the first thing to check.
• First run stalls. It's downloading gigabytes from HuggingFace. Give it a minute and watch your network or the HF cache folder.

One light caveat: this is a small, one-person pack with no real community footprint, and the README is a model list plus a VRAM apology. The code is straightforward and honest about what it does, but don't expect updates or a support forum. It works, it's local, and it's free - just treat it as the handy utility it is, not a platform.