🖥️VLM-GGUF Loader

Think of this as the GGUF Loader's vision cousin - same llama.cpp backend, same quantization tradeoffs, but built for LLaVA-family vision-language models instead of plain text LLMs. The class name (LLavaLoader) gives it away: it's specifically wired to llama.cpp's LLaVA multimodal support, which is why the configuration looks a little different from a text-only GGUF load.

The main difference from the plain GGUF Loader is that a quantized vision model needs two files, not one: ckpt_path for the main GGUF language model checkpoint, and clip_path for a separate vision-projector ("mmproj") file that handles turning image input into something the language model can attend to. This is a standard llama.cpp-ecosystem pattern for LLaVA-style models - if you've only downloaded the main GGUF file and skipped the mmproj file, this loader won't have anything usable in clip_path and you'll get an error rather than a degraded text-only mode. max_ctx, gpu_layers, and n_threads are the identical trio from GGUF Loader - context window size, how many layers get offloaded to GPU, and CPU thread count for what isn't - so the same quant-ladder logic applies: Q8 quants are close enough to full precision that most people can't tell the difference, and quality drops off noticeably from Q4 down.

The field that's genuinely specific to this loader is chat_format, and it matters more than it looks like it should. LLaVA-family models each expect their prompts formatted a particular way internally, and llama.cpp needs to know which convention to use - the choices here (llava-1-5, llava-1-6, llama-3-vision-alpha, minicpm-v-2.6, obsidian, moondream, nanollava) map to specific model families, and picking the wrong one for the model you actually loaded is a classic silent failure mode across the llama.cpp ecosystem: the model loads fine, runs fine, and just produces nonsense or ignores the image entirely because the prompt template doesn't match what it was trained on. If your VLM-GGUF setup loads without error but the output makes no sense relative to the image you gave it, this is the first field to double-check, not the model file. is_locked behaves the same as the loader nodes elsewhere in this pack - on by default, keeps the model resident so it doesn't reload every run; disable it if you're swapping models and want to release memory between loads with the pack's Clear Model node.

Installing it means installing the whole pack: search "comfyui_LLM_party" in ComfyUI Manager, or git clone https://github.com/heshengtao/comfyui_LLM_party into custom_nodes and restart, then pip install -r requirements.txt from the pack folder using ComfyUI's own Python - portable builds need python_embeded\python.exe -m pip install -r requirements.txt specifically, since the system pip won't touch ComfyUI's environment at all.

The dependency to watch is the same one as the plain GGUF Loader: llama-cpp-python, compiled and CUDA-version-sensitive. The README's own advice if a VLM or GGUF model throws an error is to grab the latest wheel from abetlen/llama-cpp-python releases rather than trust whatever pip resolved automatically - this is the most common reason a fresh install of the vision-GGUF path fails when the text-only path worked fine. And if you don't need GGUF models at all, config.ini's fast_installed flag lets you skip this dependency during setup entirely.