🧠 VRGDG General VLM 🧠

Sometimes you want an LLM to look at your frames - caption a training set, describe a reference image for a prompt, or critique a generated still. VRGDG_GeneralVLM is the pack's wrapper for that: it runs a Hugging Face vision-language model locally inside ComfyUI, with task presets that prepackage the instructions for common jobs. It's not a chat companion; it's a tool you wire into a pipeline, and it can look at up to 24 images at once.

Under the hood it inherits from the pack's Qwen-based LLM base, so it's a transformers-powered AutoModelForVision2Seq-style load with presets for google/gemma-3-4b-it, google/gemma-3-12b-it, meta-llama/Llama-3.2-11B-Vision-Instruct, or a custom model id (a Hugging Face repo or a local folder). Task presets inject a built-in instruction block for text_to_image, text_to_video, image_to_video, image_edit, captioner_training, or custom (where custom_instructions takes over).

The inputs that matter

• model_preset - which VLM. Gemma 3 4B is the fast default; 12B is better and heavier; Llama 3.2 11B Vision is the other solid option.
• task_preset - the built-in instruction bundle. captioner_training is the one you want for dataset captioning, and it's the only one that uses trigger_word (it prepends your LoRA trigger token).
• user_input - your actual task and creative direction on top of the preset.
• image_count - how many of the up-to-24 optional image1..image24 inputs show on the node. Set it to what you need; unused ones just don't exist on the graph.
• download_if_missing - lets the node fetch missing models into ComfyUI/models/LLM/Qwen. Leave on for your first run unless you manage models yourself.
• temperature / top_p / max_new_tokens - the usual sampling knobs. 0.6/0.95/800 are sane defaults.
• hf_token - needed only for gated repos like the gated Llama models; meta-llama/Llama-3.2-11B-Vision-Instruct will want a Hugging Face token with access.
• allow_custom_model_code - keep off unless a trusted custom repo (e.g. Phi-3.5 Vision) requires custom code.

Outputs: text, used_model (which model actually ran), and status.

Install

cd ComfyUI/custom_nodes
git clone https://github.com/vrgamegirl19/comfyui-vrgamedevgirl
pip install -r requirements.txt   # includes transformers, accelerate, huggingface_hub

Model weights download on first use per preset. A 4B model is a few GB; the 12B and 11B ones are chunkier, and VRAM is the binding constraint. The code already routes around known annoyances - it forces the SDPA attention implementation for Phi-3.5 Vision, for example, so that's one install fight you don't have to have.

Common issues

• Gated model access denied: set hf_token - Llama 3.2 Vision is gated and needs a logged-in HF account.
• OOM loading a 12B VLM: switch to the 4B preset, or dtype to float16/bfloat16 rather than auto.
• "Model not found" with download_if_missing off: turn it on, or place the weights in ComfyUI/models/LLM/Qwen yourself.
• Output is weirdly off-task: you've mixed a task preset and instructions that contradict it. Pick custom and own the full instruction block, or strip user_input down.

Honest take: for one-off "what's in this image" questions you'd use a chat UI. This node earns its keep in automation - captioning a dataset, feeding a description into a prompt-creator stage, validating frames mid-pipeline. That's the use case, and it's a good one.