🖥️ Local Text Model Loader (GGUF)

ComfyUI is great at generating images, but the second you want the graph to think - caption what it made, decide between two LoRAs, or write the prompt for the next step - you usually bolt on an external script. This node is the "stop doing that" answer: it loads a quantized text model into memory from inside the graph, and everything downstream just works. It's the first half of running a local LLM in ComfyUI with the ComfyUI-GGUF-VLM pack, the Qwen-focused sibling of city96's GGUF pack (see the KB's gguf panel if you want the format's history).

What it actually does

The node scans your model folders, finds every runnable .gguf text model, and hands you a dropdown. Pick one, and it returns a model_config object that a local text generation node consumes. The clever bit is in the filtering: the loader hides helper files (anything with mmproj, clip, or projector in the name) because those are vision projection files, not models. It also deliberately keeps vision-language models in the list - a Qwen2.5-VL GGUF can do plain text chat fine without its vision half, so you don't need a second file just for dialog.

The inputs that matter

Three inputs and one optional, which is a nice tight surface:

• model - the dropdown, populated from your model folders. If it says "No models found", you haven't put a GGUF where the pack looks yet.
• n_ctx - context window size. Default 8192, up to 128000. Long captions or big documents mean bumping this; the tradeoff is that a bigger context eats VRAM the whole session.
• device - Auto (detect), GPU, or CPU. Auto is usually right; pick CPU if llama-cpp keeps crashing on your card.
• system_prompt (optional) - a default persona. Or wire in the System Prompt Config node instead.

The output is one thing: model_config (TEXT_MODEL). That plugs into the pack's local text generation node. That's the entire chain - loader in, generation out.

Installing and getting models

The pack needs llama-cpp-python, and the plain pip version won't use your GPU. Install it with the CUDA wheel from abetlen's index, matching your CUDA version:

cd ComfyUI/custom_nodes
git clone https://github.com/walke2019/ComfyUI-GGUF-VLM.git
cd ComfyUI-GGUF-VLM
pip install -r requirements.txt
pip install llama-cpp-python --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu126

The README's recommended stack is PyTorch 2.9.1+cu126 with llama-cpp-python 0.3.16. Then drop .gguf files into ComfyUI/models/LLM/GGUF/ - the pack also checks text_encoders and clip folders - hit refresh, and your models appear. A Q4_K_M Qwen2.5-7B sits around 4-5 GB and runs on a mid-range card; see the GGUF quality ladder in the KB: Q8 is basically fp16 at half the size, Q4_K_M is the accepted compromise for 12 GB cards.

Where people get burned

The classic failure is the llama-cpp-python not installed error - that means you installed the CPU build, which the pack's requirements.txt pulls by default. Reinstall with the extra-index-url command above. On Windows, older llama-cpp builds also have a history of blue-screening with certain GPUs; if that happens, the pack's changelog's advice is blunt: update llama-cpp, update drivers, or drop to CPU mode and let the Memory Manager node free VRAM between runs.