A comprehensive and modular QwenVL integration for ComfyUI, providing advanced vision-language capabilities with support for both HuggingFace Transformers and GGUF models. This extension consolidates features from multiple QwenVL implementations while introducing enhanced error handling, attention backend optimization, and a clean, maintainable codebase.

Credits & Acknowledgments

This project builds upon and consolidates features from multiple excellent QwenVL implementations:

Original Implementations

ComfyUI-QwenVL by 1038lab
- GGUF backend integration
- System prompt templates
- Advanced parameter controls
- Comprehensive model support
ComfyUI_Qwen2-VL-Instruct by IuvenisSapiens
- Initial ComfyUI integration
- Multi-image and video support
- Path node implementation
- Clean workflow design

Supported Models

Qwen3.5+ Unified Vision-Language Models

Qwen3.5, Qwen3.6, and Qwen3.8 use unified thinking/instruct mode — a single model supports both reasoning and direct instruction-following, controlled by the enable_thinking toggle in the node UI. No separate -Instruct and -Thinking model files are needed.

| Model | Size | Architecture | VRAM (FP16) | VRAM (8-bit) | VRAM (4-bit) | |-------|------|-------------|-------------|--------------|--------------| | Qwen3.5-9B | 9B | Dense (Hybrid Gated Delta Net) | ~20GB | ~12GB | ~7GB | | Qwen3.5-27B | 27B | Dense (Hybrid Gated Delta Net) | ~56GB | ~30GB | ~18GB | | Qwen3.5-35B-A3B | 35B total / 3B active | MoE (Hybrid Gated Delta Net) | ~72GB | ~36GB | ~20GB | | Qwen3.6-27B | 27B | Dense unified multimodal | ~56GB | ~28GB | ~16GB | | Qwen3.6-35B-A3B | 35B total / 3B active | MoE unified multimodal | ~72GB | ~36GB | ~20GB | | Qwen3.8-27B | 27B | Dense unified multimodal | ~56GB | ~28GB | ~16GB |

FP8 variants are available for Qwen3.5-9B/27B/35B-A3B, Qwen3.6-27B/35B-A3B, and Qwen3.8-27B. The 27B FP8 entries are estimated at ~28GB and the 35B-A3B entries at ~36GB.

Note: Qwen3.5 uses a novel hybrid architecture combining Gated Delta Networks with sparse MoE. The 35B-A3B variant activates only ~3B parameters per token, making it very memory-efficient despite 35B total parameters.

Qwen3-VL / Qwen2.5-VL Models

| Model | Size | Features | VRAM (FP16) | VRAM (8-bit) | VRAM (4-bit) | |-------|------|----------|-------------|--------------|--------------| | Qwen3-VL-2B-Instruct | 2B | General VL | ~4GB | ~2.5GB | ~1.5GB | | Qwen3-VL-2B-Thinking | 2B | CoT reasoning | ~4GB | ~2.5GB | ~1.5GB | | Qwen3-VL-4B-Instruct | 4B | Balanced | ~6GB | ~3.5GB | ~2GB | | Qwen3-VL-4B-Thinking | 4B | CoT reasoning | ~6GB | ~3.5GB | ~2GB | | Qwen3-VL-8B-Instruct | 8B | High quality | ~12GB | ~7GB | ~4.5GB | | Qwen3-VL-8B-Thinking | 8B | Advanced CoT | ~12GB | ~7GB | ~4.5GB | | Qwen3-VL-32B-Instruct | 32B | Best quality | ~28GB | ~14GB | ~8.5GB | | Qwen3-VL-32B-Thinking | 32B | Complex reasoning | ~28GB | ~14GB | ~8.5GB | | Qwen2.5-VL-3B-Instruct | 3B | Previous gen | ~6GB | ~3.5GB | ~2GB | | Qwen2.5-VL-7B-Instruct | 7B | Previous gen | ~15GB | ~8.5GB | ~5GB |

FP8 Pre-Quantized Models (40-series GPU recommended):

Qwen3-VL-2B-*-FP8: ~2.5GB VRAM
Qwen3-VL-4B-*-FP8: ~2.5GB VRAM
Qwen3-VL-8B-*-FP8: ~7.5GB VRAM
Qwen3-VL-32B-*-FP8: ~24GB VRAM

GGUF Quantized Models

The unified Qwen3.5+ GGUF models are sourced from unsloth; older Qwen-VL entries retain their configured upstream repositories.

| Model | Source | Variants | Features | |-------|--------|----------|----------| | Qwen3.5 (Unified Thinking/Instruct) | | | | | Qwen3.5-9B-GGUF | unsloth | Q4_K_M, Q8_0, BF16 | Unified thinking + instruct | | Qwen3.5-27B-GGUF | unsloth | Q4_K_M, Q8_0, BF16 | Unified thinking + instruct | | Qwen3.5-35B-A3B-GGUF | unsloth | Q4_K_M, Q8_0, BF16 | MoE, unified thinking + instruct | | Qwen3.6 / Qwen3.8 (Unified Thinking/Instruct) | | | | | Qwen3.6-27B-GGUF | unsloth | IQ4/Q4/Q5/Q6/Q8, UD quants, split BF16 | Unified thinking + instruct | | Qwen3.6-35B-A3B-GGUF | unsloth | MXFP4_MOE/Q8, UD quants, split BF16 | MoE, unified thinking + instruct | | Qwen3.8-27B-GGUF | unsloth | Q4/Q8, UD quants, split BF16 | Generic MTMD, unified thinking + instruct | | Qwen3.6-27B-GGUF | lmstudio-community | Q4_K_M, Q6_K | Unified thinking + instruct | | Qwen3.6-35B-A3B-GGUF | lmstudio-community | Q4_K_M | MoE, unified thinking + instruct | | Qwen3.8-27B-GGUF | lmstudio-community | Q4_K_M, Q6_K, Q8_0 | Generic MTMD, unified thinking + instruct | | Qwen3-VL | | | | | Qwen3-VL-2B-Instruct-GGUF | unsloth | Q4_K_M, Q8_0 | Instruct tuned | | Qwen3-VL-4B-Instruct-GGUF | unsloth | Q4_K_M, Q8_0 | Instruct tuned | | Qwen3-VL-8B-Instruct-GGUF | unsloth | Q4_K_M, Q8_0 | Instruct tuned | | Qwen3-VL-4B-Thinking-GGUF | unsloth | Q4_K_M, Q8_0 | Thinking mode | | Qwen3-VL-8B-Thinking-GGUF | unsloth | Q4_K_M, Q8_0 | Thinking mode | | Qwen2.5-VL | | | | | Qwen2.5-VL-3B-Instruct-GGUF | unsloth | Q4_K_M, Q8_0 | Previous gen | | Qwen2.5-VL-7B-Instruct-GGUF | unsloth | Q4_K_M, Q8_0 | Previous gen | | Qwen2.5-VL-32B-Instruct-GGUF | unsloth | Q4_K_M, Q8_0 | Previous gen |

GGUF Quantization Guide:

Q4_K_M: ~3-4GB VRAM, good balance of quality and speed
Q8_0: ~5-7GB VRAM, better quality
BF16: near full precision and much larger than quantized variants. Qwen3.6/Qwen3.8 BF16 downloads are split into two colocated shards under BF16/; the node downloads both automatically.

Note: Qwen3.5+ GGUF support requires the JamePeng llama-cpp-python fork >= 0.3.49. Qwen3.5/Qwen3.6 use Qwen35ChatHandler; Qwen3.8 uses GenericMTMDChatHandler.

Locally mirrored LM Studio Community models use source-qualified dropdown names and are stored under models/LLM/GGUF/lmstudio-community/<repo-name>/, with each repository's matching mmproj beside its model files.

Installation

Method 1: ComfyUI Manager (Recommended)

Open ComfyUI Manager
Search for "QwenVL Utils"
Click "Install"
Restart ComfyUI

Method 2: Manual Installation

Clone this repository into your ComfyUI/custom_nodes directory:

cd ComfyUI/custom_nodes
git clone https://github.com/AkihaTatsu/ComfyUI-QwenVL-Utils.git

Install core dependencies with ComfyUI's Python interpreter:

cd ComfyUI-QwenVL-Utils
python -m pip install -r requirements.txt
python install.py

(Optional) Install optional features:

# For 4-bit/8-bit quantization
pip install bitsandbytes>=0.41.0

# For Flash Attention 2 (Ampere+ GPU)
pip install flash-attn --no-build-isolation

# For SageAttention (experimental)
pip install sageattention

# Install all optional dependencies
pip install -e ".[all]"

Restart ComfyUI

GGUF Setup (Optional)

For GGUF model support with vision capabilities:

ComfyUI Manager invokes install.py automatically. Existing compatible builds are preserved.
The installer selects CUDA on Windows/Linux, Metal on Apple Silicon, Vulkan when its SDK is detected, and CPU otherwise.
CUDA and Metal use matching pre-built wheels when available; Vulkan, CPU, and unavailable wheel combinations are built automatically from the pinned JamePeng llama-cpp-python fork.
Source builds require a C/C++ compiler and the selected backend's toolkit. The installer does not install system toolchains.
To override detection, set QWENVL_LLAMA_CPP_BACKEND to auto, cuda, metal, vulkan, cpu, or skip, then reinstall/update the node or run install.py with the ComfyUI Python.
A compatible build must be version 0.3.49 or newer and provide Qwen3VLChatHandler, Qwen35ChatHandler, and GenericMTMDChatHandler.
MTP sidecars are intentionally not loaded; these entries use standard autoregressive inference.

Node Overview

QwenVL (Basic)

Simplified interface for quick vision-language tasks. Qwen3.5+ models automatically use their recommended general-task sampling parameters; older model families retain the legacy defaults.

Parameters:

model_name: Model selection (HF or [GGUF] prefixed)
enable_thinking: Toggle Thinking/Instruct mode for unified Qwen3.5/3.6/3.8 models (see Thinking Mode)
reasoning_effort: Qwen3.8 Thinking depth (low, medium default, or xhigh); ignored by other models
quantization: Memory mode — 4-bit/8-bit/FP16 (HF only, ignored for GGUF)
attention_mode: Attention backend — auto/manual selection (HF only, ignored for GGUF)
preset_prompt: Pre-defined task prompts (❌ None sends no system instruction)
custom_prompt: Custom text prompt (replaces preset when filled)
max_tokens: Maximum new text tokens, including thinking and the final answer (64–256000). Input text and images count toward ctx separately; limited context can end generation earlier.
keep_model_loaded: Cache model in VRAM between runs
seed: Reproducibility seed

Generation Defaults (Basic node):

Qwen3.5+ Thinking: temperature 1.0, top_p 0.95, top_k 20, min_p 0.0, repetition_penalty 1.0
Qwen3.5+ Instruct: temperature 0.7, top_p 0.8, top_k 20, min_p 0.0, repetition_penalty 1.0
Older models: temperature 0.6, top_p 0.9, repetition_penalty 1.2

Inputs:

image (optional): Single image input
video (optional): Video frames sequence
source_path (optional): File path input

QwenVL (Advanced)

Full-featured node with granular control over all generation parameters.

Additional Parameters (beyond Basic):

use_torch_compile: Enable torch.compile() JIT optimization (HF only, ignored for GGUF)
device: Device override — auto/cuda:x/cpu
temperature: Sampling randomness (0.0–2.0, default: 0.6). Values <0.01 trigger greedy decoding.
top_p: Nucleus sampling threshold (0.0–1.0, default: 0.9)
num_beams: Beam search width (1–8, default: 1). Values >1 disable temperature/top_p. (HF only, ignored for GGUF)
repetition_penalty: Token repetition penalty (0.5–2.0, default: 1.2)
frame_count: Video frame sampling count (1–64, default: 16)
sampling_profile: Manual preserves the fields above; Qwen Recommended applies the model/mode preset and uses one sampling beam
reasoning_effort: Qwen3.8 Thinking depth (low, medium default, or xhigh)

HF-specific optional parameters (ignored for GGUF):

min_pixels: Minimum image resolution (default: 256×28×28 = 200,704)
max_pixels: Maximum image resolution (default: 1280×28×28 = 1,003,520)

GGUF-specific optional parameters (ignored for HF):

ctx: Context window size in tokens (0 = Auto for unified Qwen3.5+ models, range: 0–262144). Saved positive values remain explicit limits.
n_batch: Prompt processing batch size (default: 512, range: 64–32768)
gpu_layers: GPU layer offload count, -1 = all (default: -1)
image_max_tokens: Max tokens per image encoding (default: 4096)
top_k: Top-K for llama.cpp model constructor (default: 0 = disabled)
min_p: Minimum probability sampling threshold (default: 0.0 = disabled, matches HF). Higher values (e.g. 0.05) narrow token candidates and may reduce output length.
top_k_sampling: Top-K sampling during generation (default: 0 = disabled, matches HF). Non-zero values (e.g. 40) restrict each token choice to K most probable candidates.
pool_size: Memory pool size for vision processing (default: 4,194,304)

Input Utility Nodes

Load Image Advanced

Loads images with additional outputs for mask and file path.

Returns:

image: Image tensor
mask: Alpha channel mask
path: File path string

Features:

Supports animated images (GIF)
Auto EXIF orientation
Multiple image formats (JPG, PNG, BMP, TIFF, WebP, GIF)

Load Video Advanced

Loads video files from ComfyUI input directory.

Returns:

video: Video object
path: File path string

Load Video Advanced (Path)

Loads video files from custom file path string.

Input:

file: File path string (e.g., "X://path/to/video.mp4")

Returns:

video: Video object
path: File path string

Multiple Paths Input

Creates a path batch from multiple image/video files.

Parameters:

inputcount: Number of input paths (1-1000)
path_1, path_2, ...: Individual file paths
sample_fps: Video sampling FPS (default: 1)
max_frames: Maximum frames per video (default: 2)
use_total_frames: Use all video frames (default: True)
use_original_fps_as_sample_fps: Use original video FPS (default: True)

Returns:

paths: List of path objects for batch processing

Supported Formats:

Images: JPG, JPEG, PNG, BMP, TIFF, WebP, GIF
Videos: MP4, MKV, MOV, AVI, FLV, WMV, WebM, M4V

Qwen3.5+ Unified Thinking/Instruct Mode

Qwen3.5, Qwen3.6, and Qwen3.8 models embed both reasoning and direct-response capabilities in a single unified model. Instead of choosing separate files, select the mode at inference time via the enable_thinking toggle:

| Mode | enable_thinking | Behavior | Best For | |------|-------------------|----------|----------| | Thinking | ✅ ON | Model reasons step-by-step in <think>…</think> blocks before the final answer | Complex reasoning, math, code analysis, multi-step tasks | | Instruct | ❌ OFF (default) | Model responds directly without internal reasoning | Quick descriptions, tagging, straightforward Q&A |

How It Works

The enable_thinking parameter is passed to apply_chat_template(enable_thinking=True/False) for HF models
For GGUF models, the toggle is passed to Qwen35 or Generic MTMD chat-template arguments and also controls output post-processing.
When a model without the unified_thinking flag is selected, the toggle is ignored — behavior is unchanged for Qwen3-VL, Qwen2.5-VL, etc.

Sampling Recommendations

| Mode | Temperature | top_p | top_k | presence_penalty | |------|-------------|-------|-------|------------------| | Qwen3.5 Thinking (general) | 1.0 | 0.95 | 20 | 1.5 | | Qwen3.6/3.8 Thinking (general) | 1.0 | 0.95 | 20 | 0.0 | | Instruct (general) | 0.7 | 0.8 | 20 | 1.5 |

The Basic node applies these presets automatically. The Advanced node remains manual by default and applies them only when sampling_profile is set to Qwen Recommended.

Usage Guide

For High VRAM Systems (16GB+):

quantization: None (FP16)
attention_mode: flash_attention_2
keep_model_loaded: True
use_torch_compile: True

For Low VRAM Systems (<8GB):

quantization: 4-bit (VRAM-friendly)
attention_mode: auto
keep_model_loaded: False
Use GGUF models with Q4_K_M quantization

For Video Analysis:

frame_count: 16-32 (balance detail vs. speed)
max_tokens: 2048-4096 (longer outputs)

Preset Prompts

| Prompt | Use Case | Output Type | |--------|----------|-------------| | ❌ None | No system prompt | Custom only | | 🖼️ Tags | Generate comma-separated tags | Short list | | 🖼️ Simple Description | One-sentence summary | 1 sentence | | 🖼️ Detailed Description | Comprehensive paragraph | 6-10 sentences | | 🖼️ Ultra Detailed Description | Exhaustive analysis | 10-16 sentences | | 🎬 Cinematic Description | Film-style description | Atmospheric | | 🖼️ Detailed Analysis | Structured breakdown | Categorized | | 📹 Video Summary | Video content summary | Narrative | | 📖 Short Story | Creative storytelling | Fiction | | 🪄 Prompt Refine & Expand | Enhance T2I prompts | Enhanced text |

Attention Mode Selection

Auto Mode Priority

When attention_mode: auto, the system selects in this order:

Flash Attention 2 (flash_attention_2):
- Best raw performance
- Requires: Ampere+ GPU (RTX 30xx/40xx, A100, H100)
- Install: pip install flash-attn --no-build-isolation
SDPA Flash (sdpa_flash):
- PyTorch 2.0+ built-in Flash backend
- Excellent performance with better compatibility (recommended)
- Supports newer architectures (Blackwell, etc.)
- Requires: Ampere+ GPU
SageAttention (sage_attention):
- Memory efficient wrapper
- Experimental feature
- Install: pip install sageattention
SDPA Math (sdpa_math):
- PyTorch SDPA with math backend
- Slower but more stable fallback
Eager (eager):
- Standard PyTorch attention
- Always available (slowest)

Manual Selection

Force specific backend by setting attention_mode:

auto: Auto-select best available (recommended)
flash_attention_2: External flash-attn package
sdpa_flash: PyTorch SDPA with Flash backend
sdpa_math: PyTorch SDPA with math backend (disable Flash)
sdpa: Legacy option (auto-selects Flash or math)
sage_attention: SageAttention wrapper
eager: Standard PyTorch attention

Performance Benchmarks

Inference Speed (approximate)

| Model | GPU | Quantization | Tokens/sec | VRAM Usage | |-------|-----|--------------|------------|------------| | Qwen3-VL-4B | RTX 4090 | FP16 | ~120 | 10GB | | Qwen3-VL-4B | RTX 4090 | 8-bit | ~100 | 6GB | | Qwen3-VL-4B | RTX 4090 | 4-bit | ~80 | 4GB | | Qwen3-VL-8B | RTX 4090 | 8-bit | ~70 | 10GB | | Qwen3-VL-4B-GGUF | RTX 4090 | Q4_K_M | ~90 | 4GB |

Benchmarks vary based on image resolution, prompt length, and hardware.

Troubleshooting

Common Issues

GGUF context limit reached:

max_tokens limits new text only; images, video frames, and prompt text also consume ctx. Basic and Advanced ctx=0 automatically size the context for unified Qwen3.5+ models and perform one exact preflight expansion when needed. A positive Advanced ctx remains an explicit limit. If the native context or an OOM fallback still reduces the available output budget, generation returns the available text and logs the requested and effective budgets.

"Out of Memory" Error:

Solution 1: Use lower quantization (8-bit → 4-bit)
Solution 2: Use GGUF models with Q4_K_M
Solution 3: Disable keep_model_loaded
Solution 4: Close other applications
Solution 5: Use smaller model (8B → 4B → 2B)

"ImportError: transformers":

pip install "torch>=2.5.0" "transformers>=5.8.0"

Qwen3.5/Qwen3.6/Qwen3.8 fails to load (HF backend):

trust_remote_code=True is handled automatically.
Install torch >= 2.5.0 and transformers >= 5.8.0; the backend prefers AutoModelForMultimodalLM as its generic fallback.

Qwen3.5+ GGUF not loading:

Requires JamePeng's llama-cpp-python fork >= 0.3.49.
Qwen3.6 requires Qwen35ChatHandler; Qwen3.8 requires GenericMTMDChatHandler.
Run python install.py with ComfyUI's Python from this custom node directory. The installer reports the detected platform and backend.

"llama-cpp-python not found" (GGUF):

cd ComfyUI/custom_nodes/ComfyUI-QwenVL-Utils
python install.py

If automatic compilation fails, install a C/C++ compiler plus the CUDA Toolkit or Vulkan SDK requested in the installer log, then rerun the same command. Apple Silicon Metal builds require the Xcode Command Line Tools. Available pre-built wheels are published on the JamePeng release page.

"Flash Attention not available":

Requires Ampere or newer GPU (compute capability 8.0+)
Install: pip install flash-attn --no-build-isolation
Fallback: System will use SDPA automatically

Model Download Fails:

Check internet connection
Set HuggingFace token: huggingface-cli login
Manual download from Hugging Face
Place in ComfyUI/models/LLM/Qwen-VL/<model_name>/