🖥️Easy VLM-GGUF Loader

This is the node for loading a vision-language model - something LLaVA-style that can look at an image and describe or reason about it - in GGUF format, the quantized single-file format that makes running a multi-billion-parameter model on a consumer GPU actually feasible. It's the local-VLM half of the pack's "start_with_VLM_GGUF" example workflow, and it's built directly on llama.cpp's multimodal support rather than a transformers pipeline.

The two-file thing, explained

Unlike a plain text GGUF loader, a vision-language GGUF setup needs two files, not one, and that's reflected in the two required path fields here. ckpt_path is the main language-model GGUF file - the part that does the actual reasoning and text generation. clip_path is the companion vision projector file (commonly called an "mmproj" file in the llama.cpp ecosystem) - the small model that turns image pixels into the embedding space the language model understands. Both are dynamically populated dropdowns scanning wherever this pack expects model files, which means the fix for "my model doesn't show up" is almost always "the file isn't in the folder this node scans" rather than a code problem.

Get these two mismatched - a clip_path that doesn't correspond to the ckpt_path you picked - and you'll get garbage output or an outright load error, since the projector has to match the language model's embedding dimensions.

The rest of the knobs

max_ctx (256 to 128,000, default 512) sets the context window in tokens - how much conversation and image-description history the model can hold at once. The low default is conservative; bump it up if you're doing multi-turn conversation about an image rather than a single description, keeping in mind that a bigger context window costs more memory. gpu_layers (0–100, default 31) is straight out of llama.cpp: how many of the model's transformer layers get offloaded to GPU versus run on CPU. More layers on GPU means faster inference and more VRAM used; if you're running out of VRAM, this is the first dial to turn down, not max_ctx. n_threads (1–100, default 8) sets CPU threads for whatever isn't running on GPU - match it roughly to your CPU's core count, not higher. is_locked (default on) prevents the model from being reloaded every time the graph re-runs, which matters a lot for a model this size - leave it on unless you're actively swapping models and pairing it with this pack's memory-clearing node to force a fresh load.

Output is a single model object, feeding into whatever downstream node in this pack expects a loaded VLM - typically an LLM node configured for local/GGUF mode with image input attached.

GGUF, briefly

GGUF isn't unique to this pack - it's llama.cpp's own quantization format, and the Q-number in a filename (Q4, Q5, Q6, Q8) tells you how aggressively the weights are compressed. As a rule of thumb carried over from the wider GGUF ecosystem: Q8 is close enough to full precision that you generally can't tell the difference, Q5 is roughly the last stop before you start noticing quality loss, and Q4_K_M is the accepted sweet spot for constrained VRAM. For a VLM specifically, image understanding tends to degrade faster with aggressive quantization than plain text generation does, so if descriptions start getting vague or wrong at a low quant level, that's expected behavior, not a bug.

Installing it

Standard pack install - this node has no separate download:

• ComfyUI Manager: search "comfyui_LLM_party", install, restart.
• Manual: cd ComfyUI/custom_nodes && git clone https://github.com/heshengtao/comfyui_LLM_party, then pip install -r requirements.txt, restart.

The dependency that actually matters here is llama-cpp-python, and it's the single most likely thing to go wrong in this whole pack. It often needs to compile from source against your specific CUDA version, and the README itself flags this directly: if your GGUF or VLM-GGUF model throws an error, grab a prebuilt wheel from the llama-cpp-python releases page matching your Python and CUDA version rather than fighting the source build. Community reports of this exact pack failing to import after a Manager install trace back to this dependency more often than anything else - if the whole node category vanishes after install, check llama-cpp-python first.

Common issues

If the pack imports fine but this node specifically errors on load, mismatched ckpt_path/clip_path pairing is the top suspect - always use the projector file distributed alongside the specific GGUF checkpoint you're loading, not one borrowed from a different model. If you're on an older GPU or VRAM is tight, drop gpu_layers before you drop max_ctx - losing context window hurts multi-turn image conversations more than losing a bit of inference speed does.