Modern VLM (Qwen / SmolVLM2 / LFM / InternVL / Granite / Gemma)

ModernVLM is the reason people install this pack. It's the one local vision-language node that actually feels done: a deliberately small menu of twelve production models, real image and video support, live token streaming, and ComfyUI's model manager doing the memory juggling for you. If you've been chaining BLIP captions or fighting half-broken per-model VLM nodes from 2024, this is the upgrade. The name is fair - this is the modern one, and the older tiers live in a separate legacy node so they don't clutter your menu.

The picker is curated on purpose: Qwen 3 VL 2B/4B/8B Instruct, Qwen 3.5 0.8B/4B, SmolVLM2 500M and 2.2B, LFM2.5-VL 450M, InternVL 3.5 1B, Granite Vision 4.1 4B, Gemma 3 4B, or any custom Hugging Face image-to-text repo. Default is Qwen 3 VL 2B - much faster to load than the bigger checkpoints while still handling image and video understanding well. Sixteen of the choices are internally marked as the small-and-fast tier, which matters on a 6–12 GB card.

How it works: you drop in an image (or a video_frames batch, optionally with a video_selection from the pack's adaptive frame sampler), type a prompt, and the node loads the model through Transformers, runs generation, and hands you the text. It uses official model repositories with native Transformers paths rather than community quantizations - you're not gambling on an unverified conversion. temperature and top_p behave like you'd expect (0.1 / 0.9 defaults are sane for descriptive tasks). memory_mode is where the real choices live: ComfyUI managed BF16 by default, 4-bit NF4 or 8-bit via bitsandbytes, or CPU. enable_thinking is off by default for lower latency - flip it on when you actually want the model to reason before answering.

The feature people notice first is streaming. With stream_output on, tokens arrive through ComfyUI's native progress-text WebSocket, so a connected View Text node updates live and restores the last result when ComfyUI reloads your history. Disable it for headless/API runs that don't need incremental UI updates. unload_after=false caches the model per node for fast repeated queues; set it true when another big model has to run right after.

Installing it

Easiest via ComfyUI Manager - search for ComfyUI VLM nodes (pack title "VLM_nodes"). Or clone it yourself:

cd ComfyUI/custom_nodes
git clone https://github.com/gokayfem/ComfyUI_VLM_nodes
python -m pip install -r ComfyUI/custom_nodes/ComfyUI_VLM_nodes/requirements.txt

Use ComfyUI's Python for that pip call. Do not let anything replace your torch - the pack deliberately leaves the accelerator build to ComfyUI's own installer. Models download on first execution into ComfyUI/models/LLavacheckpoints, and Hugging Face downloads respect HF_TOKEN. Gemma 3 requires accepting its license on the HF page, so it'll fail with a license error until you've done that.

Common issues

• Gated model errors: Gemma 3 and PaLI-Gemma need license acceptance on Hugging Face; log in with HF_TOKEN set.
• VRAM: on a 24 GB card, Qwen 3 VL 2B is the fast default and 8B fits in BF16; for anything bigger use NF4, or stay in the legacy node where the large tiers live.
• Older installs: early versions of this pack auto-downloaded llama-cpp-python wheels on startup and Windows users hit "invalid wheel filename" errors. The current version dropped that startup installer entirely, so if you're on an old checkout, update - that whole failure mode is gone.

Wire the STRING output into the pack's Text/JSON tools, a View Text node, or straight into your workflow as the prompt source. That's the whole point: captioning, tagging, VQA, and video understanding without leaving the graph.