QwenVL (Advanced)

You've got a folder of renders that need captions, a frame you want the model to actually look at before you describe it, or a video you need summarized. That's the whole genre of "vision-language model inside ComfyUI," and QwenVL (Advanced) is the full-featured end of this pack's two-node lineup. It runs Alibaba's Qwen vision models locally - no API key, no cloud, everything on your GPU - and hands you back a plain text RESPONSE.

The catch is right there in the name: "Advanced." If this is your first VLM node, the Basic version in the same pack is the saner entry point. This one is for when you know you want control.

What it actually does

Pick a model_name and the node does the rest. First run downloads the weights from HuggingFace automatically into ComfyUI/models/LLM/Qwen-VL/ - the tooltip says it plainly: "First run downloads weights automatically." You choose between 83 models, in two flavors:

• HuggingFace (HF) models run full Transformers at FP16, or quantized to 8-bit / 4-bit via quantization (bitsandbytes). Best quality, and the only path with attention_mode and use_torch_compile.
• [GGUF] models run through llama.cpp using unsloth's quantized files. This is the low-VRAM escape hatch, and the KB's GGUF panel is the context you want: Q8 is essentially fp16 at half the size, Q4_K_M is the accepted compromise for tight cards.

One genuinely new thing worth knowing: Qwen3.5 merged thinking and instruct into a single model, so instead of picking a separate "-Thinking" file you flip enable_thinking. ON gets you step-by-step <think>…</think> reasoning; OFF (default) answers directly. It's ignored for Qwen3-VL and Qwen2.5-VL models.

The inputs that matter

Most of the 25+ knobs are set-and-forget. The ones a beginner actually touches:

• preset_prompt / custom_prompt - the built-in templates run from Tags to Ultra Detailed Description to Video Summary. Fill custom_prompt and it completely replaces the preset. This is 90% of what makes the output useful.
• max_tokens - default 512, ceiling 256000. Bump to 2048–4096 for video summaries.
• temperature (0–2, default 0.6) and top_p (default 0.9) - the usual sampling dials. Below 0.01 temperature flips to greedy decoding.
• frame_count - how many frames get uniformly sampled from a video input (1–64, default 16).
• quantization and attention_mode - HF only. attention_mode: auto picks flash-attention-2 → SDPA flash → sage → math → eager in order of what's installed. Leave it on auto.
• num_beams - HF only, default 1. Bump past 1 and it switches to beam search, which disables temperature/top_p.
• keep_model_loaded - cache the model in VRAM between runs. Nice for iterating, a trap if you're juggling memory.

The GGUF-only fields (ctx, n_batch, gpu_layers, min_p, top_k_sampling, pool_size, …) default to sane values. Change them when you know why.

How it works

Feed the optional image (an IMAGE tensor) or video (a batch of frames), pick a preset, and the node builds a chat template, applies enable_thinking through apply_chat_template(), generates, and returns RESPONSE - a single STRING you can wire into Save Text, a display node, or anything else that eats text. The source_path input is accepted but in the current release the backends consume the image/video tensors, so treat it as scaffolding for now.

Install

Via ComfyUI Manager: search "QwenVL Utils", install, restart. Or manually:

cd ComfyUI/custom_nodes
git clone https://github.com/AkihaTatsu/ComfyUI-QwenVL-Utils.git
cd ComfyUI-QwenVL-Utils
pip install -r requirements.txt

requirements.txt pulls transformers, accelerate, opencv-python, and friends. 4/8-bit needs bitsandbytes, GGUF needs llama-cpp-python (see below), and flash-attn is optional - auto mode falls back to PyTorch SDPA without it.

Where people get burned

• Out of memory. The README's ladder: drop 8-bit → 4-bit, switch to a [GGUF] Q4_K_M model, disable keep_model_loaded, or shrink the model. Also close the browser tab running three other workflows.
• Qwen3.5 GGUF won't load. It needs llama-cpp-python >= 0.3.30 with the Qwen35ChatHandler, which lives in JamePeng's fork, not stock llama.cpp: pip install git+https://github.com/JamePeng/llama-cpp-python.git
• ImportError: transformers or weird failures - make sure you're on transformers>=4.45.0; older versions miss Qwen3.5 support.
• Windows portable + attention errors. The QwenVL node family has a reported failure mode on the portable Windows build where triton tries to compile a CUDA helper at runtime and dies; a common workaround is sorting out the triton include libs or switching the attention backend.

One honest caveat: this pack is brand new - zero community footprint yet - so the above is README plus what the parent implementations (1038lab's ComfyUI-QwenVL) have taught people. Start small (Qwen3-VL-2B), get one RESPONSE flowing, then climb.