ComfyUI-FL-VLM

FL VLM

Vision-language model nodes for ComfyUI powered by Qwen3-VL. Run image and image-batch inference with full Qwen prompting (system + user), optional 4-bit/8-bit quantization, attention backend selection, and a live progress UI. The pack is structured so additional VLM families can be added without changing the workflow shape.

Features

• Qwen3-VL Backend - 2B / 4B / 8B / 32B / 30B-A3B Instruct variants, plus any custom Hugging Face model ID
• Single Image or Batch - feed a single image or an entire ComfyUI image batch
• Two Batch Modes - one combined response for all images, or one response per image
• Resize Presets - 244p / 360p / 480p / 720p tuned for Qwen's min_pixels / max_pixels budget, plus a manual mode
• Quantization - 4-bit / 8-bit via bitsandbytes for fitting larger variants into smaller VRAM
• Attention Backends - sdpa, flash_attention_2, or eager
• dtype Control - auto / bfloat16 / float16 / float32 (auto-detects bf16 support)
• Live Progress - per-stage status, token-level streaming progress, and a metadata JSON output
• VLM Model Folder - registers ComfyUI/models/VLM so HF downloads are reused across launches

Nodes

| Node | Description | |------|-------------| | FL VLM Model Loader | Downloads and loads a Qwen3-VL model. Selects quantization, dtype, attention backend, and device map. | | FL VLM Image Inference | Runs a system + user prompt against one image or a ComfyUI image batch. Outputs response and a metadata_json string. |

Installation

ComfyUI Manager

Search for "FL VLM" and install.

Manual

cd ComfyUI/custom_nodes
git clone https://github.com/filliptm/ComfyUI-FL-VLM.git
cd ComfyUI-FL-VLM
pip install -r requirements.txt

Restart ComfyUI.

Quick Start

Describe a Single Image

1. Add Load Image + FL VLM Model Loader + FL VLM Image Inference
2. Loader: pick Qwen3-VL-4B-Instruct, leave defaults
3. Inference: connect the image, leave the default prompt ("Describe the image in detail.")
4. Queue prompt - model downloads to ComfyUI/models/VLM on first run

Caption an Image Batch (one shared response)

1. Connect an IMAGE batch to FL VLM Image Inference
2. Set batch_mode to all_images_one_prompt
3. Use image_start, image_stride, max_images to slice the batch
4. Queue prompt - Qwen sees all selected images at once

Caption an Image Batch (per-image responses)

1. Same as above, set batch_mode to each_image_separately
2. The response output concatenates labeled responses ([image 0], [image 1], ...)
3. metadata_json contains a structured per-image array

Fit a Bigger Model in Less VRAM

1. Loader: pick Qwen3-VL-8B-Instruct (or larger)
2. Set quantization to 4bit (or 8bit)
3. Keep dtype at auto

Models

| Variant | HF ID | Approx. VRAM (bf16) | |---------|-------|---------------------| | Qwen3-VL-2B-Instruct | Qwen/Qwen3-VL-2B-Instruct | ~6 GB | | Qwen3-VL-4B-Instruct | Qwen/Qwen3-VL-4B-Instruct | ~10 GB | | Qwen3-VL-8B-Instruct | Qwen/Qwen3-VL-8B-Instruct | ~18 GB | | Qwen3-VL-32B-Instruct | Qwen/Qwen3-VL-32B-Instruct | ~64 GB | | Qwen3-VL-30B-A3B-Instruct | Qwen/Qwen3-VL-30B-A3B-Instruct | ~60 GB (MoE) |

Set custom_model_id on the loader to use any other Qwen3-VL checkpoint (e.g. fine-tunes from Hugging Face). Models auto-download to ComfyUI/models/VLM/ on first use.

Resize Presets

Image size strongly affects speed. The inference node resizes input via Qwen's internal min_pixels / max_pixels budget (each unit = 28 * 28 pixels).

| Preset | Approx. resolution (16:9) | Use when | |--------|---------------------------|----------| | 244p_fast | ~420x224 | Fast captions, lots of images | | 360p_balanced | ~588x336 | Default - good speed/detail tradeoff | | 480p_detail | ~840x448 | Need finer detail or small text | | 720p_high | ~1176x672 | Maximum detail (slow) | | manual_qwen_budget | uses min_pixels / max_pixels directly | You know what you're doing |

Batch Modes

| Mode | Behavior | |------|----------| | all_images_one_prompt | Sends every selected image to Qwen in a single call. One combined response. Use for "compare these N images" or "describe this sequence". | | each_image_separately | Runs the prompt independently per image and concatenates labeled responses. Use for per-image captions. |

image_start, image_stride, and max_images slice the input batch before inference.

Key Parameters

• quantization (none) - 4bit / 8bit save VRAM at a small quality/speed cost
• dtype (auto) - on high-VRAM NVIDIA GPUs, bfloat16 is usually faster than any quantization
• attention (sdpa) - flash_attention_2 is fastest where supported; eager is the safe fallback
• device_map (auto) - lets accelerate shard large models across devices
• resize_preset (360p_balanced) - main speed knob
• max_new_tokens (512) - upper bound on response length
• do_sample (false) - off for deterministic captioning; on with temperature / top_p / top_k for creative output
• seed (-1) - set >= 0 for reproducible sampling

Requirements

• Python 3.10+
• NVIDIA GPU with 8GB+ VRAM (start with 2B/4B); 24GB+ for 8B in bf16
• For flash_attention_2: flash-attn installed and a supported GPU
• For 4bit/8bit: bitsandbytes installed (already in requirements.txt)

License

Apache-2.0 - Based on QwenLM/Qwen3-VL