ComfyUI Qwen-VL LoRA

ComfyUI Qwen-VL LoRA

A ComfyUI custom node package that adds LoRA adapter support on top of Qwen-VL models for image captioning. Pairs naturally with ComfyUI-QwenVL by 1038lab — the model loader reuses its loading internals (quantization, attention backend selection, FP8 path, SageAttention patching) while outputting a QWENVL_MODEL passthrough type that flows into the LoRA loader and caption nodes.

---

Nodes

Qwen-VL Model Loader

Loads a local Qwen-VL (or Qwen3-VL) checkpoint from models/LLM/Qwen-VL/ or any directory registered under text_encoders in extra_model_paths.yaml.

| Input | Type | Description | |---|---|---| | model_name | dropdown | Model folder name, scanned automatically at startup | | quantization | dropdown | 4-bit (VRAM-friendly) · 8-bit (Balanced) · None (FP16) | | attention_mode | dropdown | auto · sage · flash_attention_2 · sdpa | | device | dropdown | auto · cuda · cpu · mps | | use_compile | bool | Enable torch.compile(mode="reduce-overhead") for extra throughput |

Output: QWENVL_MODEL — carries {model, processor} downstream.

Loading rules borrowed from 1038lab:

• BitsAndBytes (4-bit / 8-bit) and FP8 models automatically force SDPA (flash-attn is incompatible with quantized weights).
• FP8 models are loaded to CPU first with device_map=None, meta tensors are materialized (sharded checkpoints supported), then moved to the target device.
• sage mode loads with SDPA and patches attention layers afterwards via 1038lab's set_sage_attention() (requires ComfyUI-QwenVL and sageattention to be installed).

---

Qwen-VL LoRA Loader

Applies a PEFT LoRA adapter on top of any QWENVL_MODEL. Adapters are scanned from models/LLM/Qwen-VL-LoRA/.

| Input | Type | Description | |---|---|---| | model | QWENVL_MODEL | Base (or previously LoRA-patched) model | | lora_name | dropdown | Adapter folder name inside Qwen-VL-LoRA/ | | strength | float 0–2 | Scales all LoraLayer.scaling weights (1.0 = full adapter, 0.0 = bypassed) |

Output: QWENVL_MODEL — same type, LoRA weights merged into the forward pass.

Adapters can be trained with the companion train_lora.py script (Unsloth + TRL). Uses device_map=None to avoid the unhashable type: 'set' bug in accelerate's get_balanced_memory.

---

Qwen-VL Caption

Runs inference on a single image and returns a text caption.

| Input | Type | Description | |---|---|---| | model | QWENVL_MODEL | Output of Model Loader or LoRA Loader | | image | IMAGE | Standard ComfyUI image tensor [B, H, W, C] | | prompt | string | System instruction / captioning prompt (multiline) | | max_new_tokens | int 64–2048 | Maximum tokens to generate |

Output: STRING caption (also displayed inline via the ShowText node).

---

Installation

custom_nodes/
└── comfyui_qwenvl_lora/
    ├── __init__.py
    ├── nodes.py
    └── requirements.txt

cd custom_nodes/comfyui_qwenvl_lora
pip install -r requirements.txt

Dependencies: transformers, peft Optional: bitsandbytes (4/8-bit), flash-attn (Flash Attention 2), sageattention + ComfyUI-QwenVL (SageAttention patching)

---

Model directory layout

models/
├── LLM/
│   ├── Qwen-VL/
│   │   └── Huihui-Qwen3-VL-8B-Instruct-abliterated/   ← base model
│   └── Qwen-VL-LoRA/
│       └── my-lora-adapter/                             ← PEFT adapter folder

Or register extra paths in extra_model_paths.yaml:

text_encoders: /path/to/your/models/text_encoders

---

Example workflow

Model Loader → LoRA Loader → Caption → ShowText

1. Drop a Qwen-VL Model Loader node, pick your base model and quantization.
2. Connect its output to a Qwen-VL LoRA Loader, select your adapter.
3. Connect the LoRA output and a Load Image node into Qwen-VL Caption.
4. Wire the caption string to ShowText|pysssss.

---

Credits

• Attention backend selection, quantization logic, FP8 loading, and SageAttention patching are derived from ComfyUI-QwenVL by 1038lab (GPL-3.0).
• Base models: Qwen3-VL by Alibaba DAMO Academy (Apache-2.0).
• LoRA fine-tuning: Unsloth + TRL + PEFT.

---

License

GPL-3.0 — in line with the upstream ComfyUI-QwenVL license.