Start llama.cpp Router

The single-model Start llama.cpp Server is the easy path. This one is for when you want to switch between several GGUF models in one workflow without restarting the server each time - say, a small fast model for prompt rewriting and a bigger one for captioning, or a VLM for image reads. Start llama.cpp Router launches llama-server in router mode, where the server keeps a catalog of models and loads them on demand.

This is the newer, fussier mode, and the README is upfront that it needs a current llama.cpp build. Router mode depends on --models-dir and --models-max, which older builds don't expose. The pack probes the binary and refuses to launch if those capabilities are missing, so if the node just won't start, update llama.cpp first and blame the build, not yourself.

How it works

The router exposes a folder of models. The pack catalogues your configured GGUF roots, and each immediate child directory of the router's model root counts as one logical model bundle - it should contain one base model and at most one projector (for VLMs). Files sitting directly in the root are separate models. Deeper nesting is invisible to the router's one-level scan.

models_max caps how many model instances the router keeps loaded at once (default 4). With models_autoload on (the default), a prompt node's first request for an unloaded model loads it automatically. The Load Model and Unload Model nodes then give you explicit, deterministic control that waits for the router to reach the terminal loaded or unloaded state - HTTP acceptance alone isn't treated as completion.

Inputs and outputs that matter

• models_directory - (auto) picks the configured GGUF root with the most unambiguous models. If your layout is ambiguous (two base models in one folder), List Models shows you what the router actually sees.
• models_max - how many models can be resident at once.
• context_size, gpu_layers, main_gpu - same meanings as the single-model start node, applied to routed instances.
• models_autoload - off if you want models loaded only when you explicitly load them.
• sleep_idle_seconds - release a model's state after it's been idle that long; 0 disables.
• Outputs are server_url (into a prompt node) and success.

Installing

Same pack install as everything else in the suite - ComfyUI Manager, search "comfyui-llamacpp", or clone the repo and pip install -r requirements.txt. The router-specific requirement is a current llama-server with router flags, plus a sane folder layout in ComfyUI/models/LLM/gguf/:

models/LLM/gguf/
├── qwen3-8b-q4_k_m.gguf            # standalone model
└── qwen3-vl/                       # one VLM bundle
    ├── qwen3-vl-8b-q4_k_m.gguf
    └── mmproj-qwen3-vl-f16.gguf

Never mix projectors across model sizes or architectures - the router uses the bundle's pairing, and a wrong projector will make the server start and immediately exit.

Gotchas

The router is where most of the pack's sharp edges live. reload_catalog on List Models can unload changed or removed running models, and load/unload nodes wait for terminal state, so a hung router means a hung workflow - keep an eye on operation_timeout. Also, the router combines its cache with one local root; List Models is the authoritative catalog, since direct-mode dropdowns scan recursively but the router doesn't. If you mostly use one model and never switch, the single-model start node is simpler and you don't need this.