🖥️Easy LLM-GGUF Loader

This is the node behind the pack's "start_with_LLM_GGUF" example workflow, and it's worth knowing that GGUF isn't a format this pack borrowed for LLMs - it's the format's actual home turf. GGUF came out of the llama.cpp project specifically for running quantized language models efficiently on consumer hardware, before the image-generation world adopted the same idea for diffusion models years later. So everything you might already know about GGUF quant tiers from the image-generation side applies here at least as directly, if not more so.

Inputs

model_path is a dynamic dropdown scanning for GGUF files in wherever this pack expects local models to sit - if a file you've downloaded isn't showing up, it's not in the right folder, not a code problem. max_ctx (256–128,000, default 512) is the context window in tokens; the default is small, appropriate for a quick single-turn response but limiting for anything with real conversation history or long documents fed in - raise it if you need the model to remember more, keeping in mind that context length costs memory. gpu_layers (0–100, default 31) is the classic llama.cpp lever: how many of the model's layers run on GPU versus CPU. Set it higher for speed if you've got the VRAM, lower it as your first move if you're running out of memory. n_threads (1–100, default 8) sets CPU threads for whatever isn't offloaded to GPU - there's little benefit setting this meaningfully above your actual CPU core count. is_locked (default on) keeps the model resident instead of reloading it every graph run, which matters a lot for anything multi-gigabyte; turn it off only when you're actively swapping models and pairing it with this pack's memory-clearing node.

Output is a single model object, feeding into this pack's LLM node configured for local/GGUF inference.

Why GGUF, and which quant to pick

The whole appeal is running a model that wouldn't otherwise fit. A 7-8B parameter model at full precision needs roughly 14-16GB just for the weights; the same model quantized to Q4_K_M drops to somewhere around 4-5GB, at some cost to output quality. The rule of thumb that's held up across the GGUF ecosystem generally: Q8 is close enough to full precision that most people can't tell the difference in a blind comparison, Q5 is roughly the last stop before quality loss becomes noticeable, Q4_K_M is the generally accepted sweet spot for 8-12GB cards, and Q3 and below are for genuinely VRAM-starved setups where "it runs at all" beats "it's good." If you've got the VRAM headroom, there's no real reason to pick a lower quant than Q8 - the size savings below that point start costing more in quality than they're worth.

Installing it

• ComfyUI Manager: search "comfyui_LLM_party", install, restart.
• Manual: cd ComfyUI/custom_nodes && git clone https://github.com/heshengtao/comfyui_LLM_party, then pip install -r requirements.txt, restart.

llama-cpp-python is the dependency doing the actual work here, and it's the one most likely to cause install pain - it frequently needs to be compiled against your specific CUDA setup, and a source build can fail in ways that are hard to debug from the pip output alone. The README addresses this directly: if a GGUF or VLM-GGUF model throws an error, grab a matching prebuilt wheel from the llama-cpp-python releases page rather than fighting the source install. People have reported this whole pack failing to import after a clean Manager install with a working PyTorch/CUDA setup otherwise - llama-cpp-python compilation is the first place to look when that happens, not your CUDA install overall.

If you know you'll never touch GGUF models, config.ini has a fast_installed option - set it to True to skip this dependency's heavier install path entirely and speed up setup.

Common issues

Out-of-memory errors: drop gpu_layers before you drop max_ctx - losing GPU offload costs speed, losing context window costs the model's actual working memory, and the latter usually hurts output quality more. If the node loads but generation is painfully slow, check whether gpu_layers is actually landing on GPU at all - a llama-cpp-python build that compiled without CUDA support silently falls back to CPU-only inference, which will run but at a fraction of the speed, with no obvious error telling you why.