LLaVA Vision Projector Loader

Here's the thing nobody warns you about with LLaVA in ComfyUI: it's not one model, it's two files. A LLaVA GGUF (the language model, e.g. ggml-model-q4_k.gguf) plus a vision projector (mmproj-model-f16.gguf) that turns images into tokens the language model can read. LlavaClipLoader is the node that loads that second file - the clip projector - into a reusable CUSTOM handle so the LLaVA loader can combine the pair.

Why you need it

Because the projector is not optional and not interchangeable. Each LLaVA-family model ships its own mmproj tuned for that exact checkpoint - the README is blunt: "every model's clip projector is different." Pair a Mistral-7B LLaVA GGUF with a 1.5/1.6 projector and you'll get garbage. Loading the projector as its own object is also what makes the modern, composable loader flow work.

How it works

The input clip_name is an enum populated from the files in your ComfyUI/models/LLavacheckpoints folder - the pack registers that folder on import, so drop mmproj-model-f16.gguf in there and it appears in the dropdown. The node returns a clip CUSTOM handle. Current versions also expose a handler optional input that picks how the projector is interpreted: the default metadata-driven "Auto (GGUF chat template)" handles most modern GGUF + mmproj pairs, with named handlers for older LLaVA 1.5/1.6-style model cards that need explicit treatment.

The two-file workflow

The intended graph looks like:

1. LlavaClipLoader - pick your mmproj file, get a clip.
2. LLava Loader (LLava Loader Simple) - pick the GGUF, wire the clip in, get a model.
3. LLaVA Sampler - feed it an image + prompt + that model, get text.

If you'd rather not juggle two loaders, the pack's LLava Optional Memory Free Simple node loads both files itself in one step - but it's a chunkier node, and the two-loader path is the classic one most old workflows use.

Install

Same pack, same drill:

cd ComfyUI/custom_nodes
git clone https://github.com/gokayfem/ComfyUI_VLM_nodes
python -m pip install -r ComfyUI/custom_nodes/ComfyUI_VLM_nodes/requirements.txt

Common issues

The #1 mistake is mixing projectors across model versions - double-check you downloaded the mmproj that matches your GGUF's model family. The #2 mistake is expecting the dropdown to show files you placed before restarting ComfyUI; the folder scan happens at load time, so restart after adding files. And like every GGUF node in this pack, it rides on llama-cpp-python, so if loading errors out, the fix is usually installing a llama.cpp wheel built for your GPU backend rather than the default CPU build.