[llama.cpp] Hardware Runtime Profile

This is a pure plumbing node. It has no inputs that come from anywhere, one output, and it does nothing on its own - it just bundles seven llama.cpp runtime settings into a single typed cable you plug into [llama.cpp] Create Runtime Session.

Which raises the fair question of why it exists. Two answers. First, the same settings reused across several session nodes become one thing to edit instead of five. Second, and more practically, this is where you go when the model is the thing that's filling your card, because the knobs here are exactly the ones that trade memory for context length and prompt-processing throughput. The pack's shipped example subgraph wires one in and sets both cache types to Q8_0 rather than the default FP16, which tells you what it's for.

How it works

Each field maps to a llama.cpp server argument, and the node validates the combinations before they ever reach the process - gpu_layers has to be one of all, auto, cpu; cache types are restricted to the three llama.cpp actually supports. The output is a typed OLLAMA_IMAGE_LIST_LLAMA_CPP_HARDWARE_RUNTIME_PROFILE handle; the session node reads it at creation time. A valid profile is required to be complete, so a half-populated one fails loudly rather than defaulting the missing bits quietly.

Note the ordering: connection is not live. If you edit the profile after the session node has run, you need to re-queue for the server to be rebuilt with the new settings.

The inputs that matter

gpu_layers - auto, all, or cpu. auto lets llama.cpp decide; all pushes every layer to the GPU (and is required if you're using the native MTP speculation path); cpu is for when you've deliberately decided the LLM should live in system RAM and leave the card alone. If a model is spilling into shared memory and crawling, all vs auto is the first thing to try.

type_k / type_v - the key and value cache data types: FP16, Q8_0, Q4_0. This is the single most useful field on the node. The KV cache grows with context length, so at a long n_ctx a quantised cache is often the difference between "8k context fits" and "8k context OOMs". Q8_0 halves cache memory with a quality hit most people can't hear; Q4_0 squeezes harder and does start to show.

flash_attention - auto, enabled, disabled. On long contexts this is usually free speed and less memory. auto is a fine default; enabled is worth trying before you conclude your card can't hold the context you want.

main_gpu - which GPU indexes as primary in a multi-GPU box. 0 unless you have a reason.

n_threads - CPU threads for the work that stays on CPU. 0 means let llama.cpp pick. Only worth setting if you've got a busy CPU or a weird core layout.

use_mmap - memory-map the GGUF instead of reading it in. true is the default and it's usually better: the OS pages in what's needed, so a model that's bigger than your free RAM can still start. Set it to false if the file lives on a network share or a slow external drive where mapping behaves badly, or if you want the model fully resident and consistent.

Output: hardware profile, one cable, into Create Runtime Session.

Install

Manager → search llama multimodal → ComfyUI-llama-multimodal. Or:

cd ComfyUI/custom_nodes
git clone https://github.com/craftingmod/ComfyUI-llama-multimodal.git

ComfyUI 0.19.3+. This node needs no model of its own; it's only useful alongside a Runtime Session, which needs the llama executable - grab it from Settings → llama-multimodal → Download llama.cpp (Windows/Linux CPU, CUDA or ROCm builds, macOS arm64 CPU) or put your own on PATH. GGUFs and their mmproj go in ComfyUI/models/LLM.

Where people get burned

Wrong mental model: this doesn't make a slow model fast. GPU layers and cache quant change what fits. A 20B model that's swapping because it doesn't fit will still be slow at gpu_layers: all - it'll just be swapping less. Pick a quant you can actually fit, then use these settings to buy back context.

Quantised cache has a ceiling. Q4_0 K/V with a very long context can produce drift and repetition that looks like a bad model. If quality fell off a cliff after you tuned this node, put type_k/type_v back to FP16 before you blame the GGUF.

cpu for everything is slower than you think. Useful to protect a small card for the diffusion model, but expect a large generation time penalty. Time one generation before you build a workflow around it.

Settings only take effect on a fresh session. Edit, then re-queue the whole thing - a warm server won't retroactively change its cache type.