Load Vision Model

"Load Vision Model" is the front door to the whole SeanScripts pack, and the only reason most people find the rest of it. Where the pack's other loaders only show models of a single architecture, this one lists every folder you drop into ComfyUI/models/LLM, reads the model's config.json to figure out what kind of transformer it is, and hands you a VISION_MODEL object ready for any of the generation nodes.

The name is a lie in the best way: it doesn't call any API and needs no key. Everything runs locally through HuggingFace transformers, the same way your checkpoints do. The "auto" means it auto-detects the architecture - Pixtral, Llama 3.2 Vision, Molmo, whatever has a valid config - instead of making you pick the right loader.

How it works

When the node opens, it scans ComfyUI/models/LLM for subfolders containing a config.json, then reads the architectures field and matches it against a small built-in map: LlavaForConditionalGeneration → Pixtral, MllamaForConditionalGeneration → Llama Vision, MolmoForCausalLM → Molmo, with a generic fallback for anything else. Loading is lazy, which is a nice touch: this node only loads the processor/tokenizer. The multi-GB weights don't load until the generate node actually runs, so wiring up a big workflow is cheap until you execute it.

Inputs and output

Two inputs, both obvious:

• model_name - the dropdown of every model folder found in models/LLM.
• trust_remote_code - boolean, default false. The author's own warning is "use at your own risk," and given this ecosystem's history with malicious custom nodes, defaulting to off is the right instinct. Only flip it for models you actually trust; note that some models (Molmo included) genuinely need custom code to load.

Output is a single VISION_MODEL that plugs into Generate Text with Pixtral/Llama Vision/Molmo.

The catch: because it lists everything with a config.json, you'll see plain text LLMs in the dropdown too. The README is blunt that non-vision models - Florence2 and friends - won't work with these generation nodes. The strict per-model loaders exist to avoid exactly that.

Install

Install once, use everywhere. ComfyUI Manager knows this pack as ComfyUI-PixtralLlamaVision:

cd ComfyUI/custom_nodes
git clone https://github.com/SeanScripts/ComfyUI-PixtralLlamaMolmoVision

Then make sure transformers >= 4.45, accelerate, bitsandbytes, and torchvision are current (Manager installs them automatically; you may also need a newer PyTorch). Models live in ComfyUI/models/LLM, one folder per model:

ComfyUI/models/LLM/pixtral-12b-nf4/
├── config.json
├── model.safetensors
├── tokenizer files...

SeanScripts publishes 4-bit quantized Pixtral and Llama Vision builds on HuggingFace that fit in normal VRAM.

Common issues

• Empty dropdown - the model folder isn't in models/LLM, or it's missing config.json. The node only picks up folders with a config.
• Generation fails with a non-vision model - you picked a plain LLM from the auto list. Use the specific loaders.
• trust_remote_code errors - a model that needs custom code was loaded with it off. Flip the toggle, or use the Molmo loader which forces it on.
• VRAM - a 12B model in full precision will choke an 8GB card; use the nf4 builds.

Start here if you're not sure which of the three model families you want - then switch to the dedicated loader once you know.