ComfyUI Node
Start llama.cpp Router
Launches and owns llama-server in multi-model router mode for explicit, terminal model load and unload control.
Start llama.cpp Router
- server_url
- success
◄context_size4096►
◄gpu_layers►
◄main_gpu0►
◄models_max4►
◄port8080►
◄threads►
◄batch_size512►
◄flash_attentionfalse►
◄models_autoloadtrue►
◄timeout60►
◄binary_path►
◄host127.0.0.1►
◄tensor_split►
◄no_mmapfalse►
◄flash_attention_modelegacy►
◄sleep_idle_seconds0►
◄api_key_file►
◄api_key_envLLAMACPP_API_KEY►
◄media_path►
◄fit_modeupstream default►
◄unload_comfy_models_before_startfalse►
◄extra_args►
◄models_directory(auto)►
CategoryLlamaCpp
Inputs (23)
| Name | Type | Default | Description |
|---|---|---|---|
| context_size | INT | 40960–1048576 | Maximum context size for routed models. 0 uses upstream logic. |
| gpu_layers | STRING | Layers to offload to GPU. Empty preserves legacy all-layers behavior; auto, all, or a number are also accepted. | |
| main_gpu | INT | 00–31 | Primary GPU index used by routed model instances. |
| models_max | INT | 40–128 | Maximum model instances the router may keep loaded. |
| portopt | INT | 80801–65535 | TCP port for the owned llama-server router. |
| threadsopt | STRING | CPU generation threads. Empty lets llama-server choose. | |
| batch_sizeopt | INT | 5121–65536 | Logical prompt-processing batch size for routed models. |
| flash_attentionopt | BOOLEAN | false | Legacy flash-attention toggle. Prefer flash_attention_mode for new workflows. |
| models_autoloadopt | BOOLEAN | true | Load an unloaded router model automatically on first request. |
| timeoutopt | STRING | 60 | Router startup readiness deadline. Empty has no limit. |
| binary_pathopt | STRING | Explicit llama-server executable. Empty uses LLAMA_SERVER_BINARY, then PATH. | |
| hostopt | STRING | 127.0.0.1 | Network interface address to bind. Loopback is safest. |
| tensor_splitopt | STRING | Per-GPU model proportions, for example 3,1. | |
| no_mmapopt | BOOLEAN | false | Disable memory-mapped model loading. |
| flash_attention_modeopt | COMBO | legacy | Modern flash-attention mode. Legacy preserves the released flash_attention widget behavior. |
| sleep_idle_secondsopt | INT | 00–86400 | Release each model's state after this idle period. 0 disables. |
| api_key_fileopt | STRING | Path to a llama-server API key file, one key per line. | |
| api_key_envopt | STRING | LLAMACPP_API_KEY | Environment variable containing the matching client key. The secret is not serialized. |
| media_pathopt | STRING | Directory allowed for llama-server file:// media inputs. | |
| fit_modeopt | COMBO | upstream default | Control whether llama-server adjusts unset arguments to fit device memory. |
| unload_comfy_models_before_startopt | BOOLEAN | false | Evict Comfy-managed models before router startup. |
| extra_argsopt | STRING | Advanced llama-server arguments. No shell is used. | |
| models_directoryopt | COMBO | (auto) | Configured GGUF root exposed to the router. Auto selects the root with the most unambiguous models visible to llama-server's one-level directory scan. Each bundle directory must contain one base model and at most one projector. |
Outputs (2)
| Name | Type | Description |
|---|---|---|
| server_url | STRING | URL of the running owned llama-server router. |
| success | BOOLEAN | Whether router startup reached its ready state. |