Qwen-VL LoRA Loader

Here's the thing about this pack: ComfyUI already has ways to run Qwen-VL. The upstream it's built on, 1038lab's ComfyUI-QwenVL, loads the models and captions images fine. What it doesn't do is apply LoRA adapters. That's the entire reason Dangocan's fork exists, and Qwen-VL LoRA Loader is the node that delivers on it.

In diffusion land you know LoRAs as small 10–200 MB files that add a style or a character to a checkpoint. Same idea, different currency here: these are PEFT adapters - a folder of weights trained on top of a specific vision-language model, changing how the model captions. You train one with the companion train_lora.py script (Unsloth + TRL under the hood), or download one someone already trained, drop it in, and your captioner now describes things the way you want it to.

How it works

The node takes a QWENVL_MODEL (from the pack's Model Loader, or from a previous LoRA loader - the output is the same type, so you can chain them) and wraps it with Hugging Face's PeftModel. Two implementation details are worth knowing because they're the difference between "works" and "errors":

• It calls PeftModel.from_pretrained(base, lora_path, device_map=None) deliberately. device_map=None sidesteps a known accelerate bug - unhashable type: 'set' in get_balanced_memory - that bites when you hand PEFT a device_map="auto" on a quantized model.
• It doesn't just load the adapter; it walks every LoraLayer and rewrites the scaling weights to your strength value, then puts the model in eval mode. That's the whole strength mechanism, applied at load time.

The inputs that matter

• model - a QWENVL_MODEL. This is a required input; a LoRA is a patch, not a model, so there's nothing to load without a base.
• lora_name - a dropdown of adapter folders inside models/LLM/Qwen-VL-LoRA/. It's scanned at startup, so a fresh folder needs a ComfyUI restart to show up. "(none)" means you haven't put one there yet.
• strength - float from 0.0 to 2.0, default 1.0. Here 1.0 is the full adapter as trained; 0.0 bypasses it entirely. Unlike diffusion LoRAs where 0.5–0.8 is the common sweet spot for stacking, this is usually fine at 1.0 - but if the adapter's captioning style is too dominant, dialing down to 0.6–0.8 blends it with the base model's behavior.

The output is a QWENVL_MODEL of the same type, so the natural graph is Loader → LoRA Loader → LoRA Loader (stacking adapters works) → Caption.

Installing and using it

The pack installs in one step - Manager search "ComfyUI Qwen-VL LoRA", or:

cd ComfyUI/custom_nodes
git clone https://github.com/Dangocan/comfyui_qwenvl_lora
cd comfyui_qwenvl_lora
pip install -r requirements.txt

(peft and transformers are the only hard requirements.) Adapters live at models/LLM/Qwen-VL-LoRA/<name>/, each a folder with the PEFT files inside. If you train with the included script, it'll already be in that shape.

Where people get burned

The classic one: this is not the diffusion-style .safetensors LoRA you download from CivitAI. Those target the UNet/text-encoder of an image model and won't load here at all - PEFT needs a folder with adapter_config.json plus adapter weights. If the dropdown stays empty, that's the disconnect.

Second, an adapter is architecture-bound, and unlike diffusion models you can't paper over it: a Qwen2.5-VL adapter dropped onto a Qwen3-VL base will fail or silently misbehave. Train or download for the exact base you're loading.

Third, match the base's format. If your base is quantized 4-bit, PEFT loading can get grumpy - keep the adapter and base on the same architecture and it's usually fine, but expect some fiddling the first time. The strength slider goes to 2.0, but pushing past ~1.2 on a captioning adapter tends to overdrive it into degenerate output; that's a trap, not a feature.