Qwen2-VL

If you've bounced off LLaVA captioning because the descriptions come out vague, this is the node to try. Qwen2-VL is Alibaba's current-generation vision-language model, and it's a different class of thing from the 2024-era LLaVA GGUFs: it reads text in the image, understands layouts, answers questions in several languages, and handles mixed resolutions without you resizing anything first. Inside gokayfem's VLM Nodes pack, the Qwen2VLNode is the flagship modern VLM, and for most people it's the whole reason to install the pack.

What it's for

Reach for it whenever you need a detailed, reliable description of an image: recaptioning a training set (captioning quality matters more than dataset size), generating alt text at scale, or asking pointed questions - "what does the label on this bottle say?", "how many people are in frame?". You get one text out of it, but the quality bar is high enough that it's the node people actually keep in production workflows.

How it works

On first run the node downloads the model you picked from Hugging Face into ComfyUI/models/LLavacheckpoints/files_for_qwen2vl and runs it through Transformers. It's a full checkpoint load, not a llama.cpp GGUF like the older LLaVA nodes, so it uses ComfyUI's own VRAM budget and bf16 inference on GPU. The model_name menu offers 2B, 7B, and 72B, plus AWQ/GPTQ-labeled variants - those quantized labels are resolved to bitsandbytes 4/8-bit loading in current versions, which is a good thing because it stopped breaking across Transformers releases.

The inputs that matter

• model_name - start with Qwen2-VL-2B. It's shockingly capable for its size. Only go to 7B if you need better OCR or reasoning; the 72B choices are for people with a GPU you don't have.
• memory_mode - Balanced (8-bit) or Maximum Savings (4-bit) when VRAM is tight, CPU Offload when it's really tight.
• text_input - the instruction. "Describe this image in detail" works; adding "mention any text you can read" gets you the OCR behavior.
• max_new_tokens - 512 default; long captions or questions with context deserve more.

The two optional inputs, video_frames and fps, let you hand it a batch of frames (say, one per second from a video) and get a temporal description. That's a neat trick that few of the pack's other nodes pull off.

Install

ComfyUI Manager - search VLM_nodes - or the manual route:

cd ComfyUI/custom_nodes
git clone https://github.com/gokayfem/ComfyUI_VLM_nodes
python -m pip install -r ComfyUI/custom_nodes/ComfyUI_VLM_nodes/requirements.txt

Use ComfyUI's Python for the pip step, restart, and give the first run time to pull the checkpoint. The pack requires Python 3.9+.

Where people get burned

• The 7B and 72B models are heavy. 7B wants roughly 16GB of VRAM unquantized; the pack estimates 145GB for the 72B, so the 4-bit mode is not optional there. If you're on 8GB, stick with 2B and maybe Maximum Savings (4-bit).
• The first run downloads gigabytes and can look hung. It isn't; watch the console.
• 72B is a trap for most people. It's a real model, it's just not a real option on consumer hardware.

Feed the STRING output into the pack's ViewText to read it, or straight into a prompt generator to turn the caption into a usable generation prompt.