LLaVA Loader

This is the node that started the pack. gokayfem's VLM Nodes began life as one of the cleanest ways to run LLaVA captioning inside ComfyUI, and LLava Loader Simple (registered as class LLava Loader Simple, displayed as "LLaVA Loader") is the loader half of that original flow. It takes the LLaVA GGUF - the language-model file - and pairs it with a vision projector you've loaded separately, producing the model handle that the LLaVA sampler consumes.

What it's for

Building the classic captioning chain: load LLaVA → describe an image → use the caption. If you've seen old workflows that caption an image and feed the text into a prompt, this is the loader they used. It's still kept working for exactly that reason: thousands of saved graphs depend on it.

How it works

LLaVA under llama.cpp is a two-file system: the main GGUF (ggml-model-q4_k.gguf-style) plus an mmproj-model-f16.gguf vision projector that converts images into tokens. This loader handles the GGUF; the LlavaClipLoader handles the projector. You pick ckpt_name (GGUF), wire in the clip you loaded, and get a model handle back. Importantly, the actual model bytes are only loaded when a downstream sampler executes - the loader itself is lazy, which keeps it from hogging VRAM while you build the graph.

The inputs that matter

• ckpt_name - the GGUF in models/LLavacheckpoints.
• clip - the CUSTOM output of LlavaClipLoader. The projector has to match the GGUF's model family.
• max_ctx - context, 4096 default; fine for one-shot captioning.
• gpu_layers - GPU offload; 27 suits a 7B. Raise toward the max if you have VRAM.
• n_threads - CPU threads for the layers that stay on CPU.

Output: a CUSTOM model handle → LLaVA Sampler (LLavaSamplerSimple) with an image and prompt.

Install

cd ComfyUI/custom_nodes
git clone https://github.com/gokayfem/ComfyUI_VLM_nodes
python -m pip install -r ComfyUI/custom_nodes/ComfyUI_VLM_nodes/requirements.txt

Plus the llama-cpp-python backend wheel matching your GPU. The original README pointed at specific GGUF sources - LLaVA 1.6 Mistral, Nous Hermes 2 Vision, LLaVA 1.5 7B/13B, BakLLaVA - and that advice still holds: any LLaVA-family GGUF whose projector you also have will work.

The tradeoff

The current README steers new graphs toward the pack's newer all-in-one LLava Optional Memory Free Simple or the Modern VLM interface, and honestly, for a fresh workflow those are less ceremony. But this two-loader path is the one you'll find in old saved workflows, it still runs, and it gives you a reusable model handle you can share between samplers. If you're debugging an old workflow that says it's missing "LLava Loader Simple", it's this pack - install VLM Nodes and the loader comes back.