FL VLM Model Loader

FL VLM Model Loader is the unglamorous half of a two-node pack, and it handles the part that decides whether your workflow lives or dies on the first click: getting a Qwen3-VL model into VRAM and handing it to the inference node as a single FL_VLM_MODEL. Nothing in the pack runs without it, and when something goes wrong, it usually goes wrong here first. The other node - FL VLM Image Inference - does the actual reading; this one is the engine, the download, and the memory budget you have to manage.

Why bother running a VLM inside the graph at all? Because Qwen3-VL is the current heavyweight among open vision-language models, and running it in ComfyUI instead of in an external LM Studio or Ollama server keeps VRAM under ComfyUI's control. That's a real community pattern - people run the 8B alongside a diffusion model on a single card precisely because the graph can juggle the memory instead of an LLM server hoarding it.

How it works

Under the hood it resolves a Hugging Face repo id (your custom_model_id overrides the dropdown), downloads the snapshot into ComfyUI/models/VLM on first use, then loads it with Qwen3VLForConditionalGeneration and the matching processor. Quantization is handled by bitsandbytes, attention by the backend you pick, and the loaded model gets cached by a key of model + dtype + quantization + attention + device map, so re-queuing a workflow doesn't reload gigabytes you already have in memory.

The inputs that actually matter

• model_variant - the dropdown of the five Qwen3-VL Instruct sizes (2B, 4B, 8B, 32B, and the 30B-A3B MoE). Defaults to 4B, which is a sane starting point.
• custom_model_id - paste any Hugging Face id to load a fine-tune instead of a stock variant. Leave empty unless you need it.
• quantization - none, 8bit, or 4bit. This is the "fit a bigger model in less VRAM" lever. Costs a little quality and speed.
• dtype - auto is the right default: it reads your GPU and picks sensibly. bfloat16 appears in the list when your card supports it; on cards that don't, you just get float16/float32.
• attention - sdpa by default and fine for most people. flash_attention_2 is fastest where supported, but requires a separately-installed flash-attn. eager is the safe fallback.
• device_map - auto lets accelerate shard big models across devices; cpu is there if you're desperate.
• force_reload - bypass the model cache when you've changed something upstream or a download got corrupted.

The single output is vlm_model, which you wire straight into the inference node's vlm_model input. There's nothing else to it.

Installing it

This is a small, fresh pack (v0.1.x at the time of writing), so installation is the standard dance:

cd ComfyUI/custom_nodes
git clone https://github.com/filliptm/ComfyUI-FL-VLM.git
cd ComfyUI-FL-VLM
pip install -r requirements.txt

Or just search "FL VLM" in ComfyUI Manager and let it handle the requirements. Either way, restart ComfyUI afterward. Note the real dependency list: transformers>=4.57, accelerate, qwen-vl-utils, bitsandbytes - this is not a node that works out of the box with a bare ComfyUI install, which is exactly why Manager matters.

Gotchas

First run downloads the model, and it's a chunky download. Budget roughly 6 GB VRAM for the 2B, 10 GB for the 4B (the default), and ~18 GB for the 8B in bf16 - the README is honest that you want 24 GB+ to be comfortable with the 8B. If you're on an 8 GB card, start at 2B or 4B and only reach for quantization when a bigger model genuinely needs it, because dtype: auto on a modern NVIDIA card is usually faster than any quantization. The flash_attention_2 option silently won't be great if flash-attn isn't actually installed, so don't chase it until you've confirmed.

One thing worth knowing: the author, filliptm, runs the well-known FL family - Fill-Nodes, FL-Trainer, FL-Qwen3TTS - so this isn't an anonymous script, and the pack is Apache-2.0. Still, this category ("LLM vision" node that downloads its own weights) is the exact shape of the one node in ComfyUI history that shipped malware, so the house rule applies: install through Manager, glance at the source before the first run. It's two small files.