llama.cpp Generate

What it actually is

The thing to understand before anything else: your LLM never runs inside ComfyUI. llama.cpp Generate talks to a llama-server process that the rest of this pack started for you, over HTTP, and streams the text back. ComfyUI stays the driver; llama.cpp is the engine in the next room. No cloud service, no API key, no bill per call.

Why bother instead of just typing a prompt? Because a GGUF chat model in your graph does jobs diffusion can't: rewriting "moody portrait, rainy alley" into the structured 60-word prompt your Z-Image or Flux checkpoint actually responds to, or looking at an image and describing it so you can img2img it. The community has wanted this without a separate Ollama daemon for a while - this pack is a fairly direct answer, with the LLM and the diffusion model taking turns on one GPU instead of fighting over it.

The turn-taking is the real feature. An 8B model at Q4 eats five-ish GB of VRAM it has no business holding while you're sampling. Start Server loads it, Generate uses it, Release After Generation gives the card back, and the pack hooks ComfyUI's own Unload Models button so the thing you already click frees the LLM too.

This is the pack's canonical generation node - the author's own recommendation for new workflows. The older Basic / ADV / ADV++ prompt nodes still work for saved graphs, but they're the legacy surface now.

How it works

Your prompt, system prompt, profile, images and any earlier messages get assembled into one chat request, POSTed to the running server, and read back as a stream - so the node can show the response live and the Stop button can actually cancel something.

Completion is explicit, which is the design decision worth understanding. A generation counts as successful only if the stream ends with a valid terminal marker. Text from a timeout, a cancellation or a broken connection is kept but labelled partial, and the default raise_error policy makes the node throw rather than hand you half a sentence that quietly flows into your CLIP encoder. That's strict on purpose, and it's the right default.

Inputs you'll actually touch

prompt is the only required one, and system_prompt is where you put your standing instruction ("you write photographic prompts, 60 words, no preamble").

For vision, set image_amount from 0 to 10 - that's how many image_1…image_10 sockets appear. All ten sockets always exist under the hood so saved workflows survive a reload, but you only see the ones you asked for. include_image_batch decides whether a whole batch goes in one request or just the first frame.

max_tokens is your output ceiling (2048 default) and request_timeout is the total deadline for the request plus any release wait. Leave sampling_mode on default - that sends no sampler settings at all, so the model keeps whatever llama.cpp and its author intended. Switch to custom and every temperature/top_p/top_k/min_p/penalty field gets sent together, which is how people accidentally wreck a model's output with a temperature it was never tested at. thinking_mode on auto likewise sends no override.

Two worth knowing: messages takes an explicit earlier-turns history (from the Append Message or Messages from JSON nodes) - cache_prompt is a prefix cache, not memory, so don't expect a chat. And structured_output accepts a JSON / JSON Schema / GBNF constraint if you need parseable text instead of prose. server_url is advanced and usually you leave it empty to mean "the server I started"; filling it and wiring connection is rejected before anything runs.

Outputs

response is the text. thinking is the reasoning stream, reported separately, and empty on models that don't have one - that's not a bug. result is the typed generation result; wire it to llama.cpp Result JSON when you want to see completion state, usage, seed and release evidence instead of trusting the preview text.

Installing it

Search "comfyui-llamacpp" in ComfyUI Manager, or:

comfy node install comfyui-llamacpp

Then the part people skip: the LLM runtime is not a pip dependency. The pack's requirements.txt is just aiohttp, numpy, pillow, psutil and requests - nothing that compiles. You supply a llama-server build yourself, from the llama.cpp releases (CUDA, Vulkan, ROCm, Metal). On Windows, extract both the binaries archive and the matching CUDA runtime archive into the same folder; copying only llama-server.exe over an older install is the classic breakage now that companion DLLs ship alongside it.

Drop at least one GGUF in ComfyUI/models/LLM/gguf/, restart, and run Workflow > Browse Templates > comfyui-llamacpp > Setup Check. Server Status will say Setup: ready or tell you exactly what's missing. If the binary lives somewhere odd, set LLAMA_SERVER_BINARY or point the Start node's binary path at it - resolution goes input, then env var, then PATH.

From source it's the usual:

cd ComfyUI/custom_nodes
git clone --branch master https://github.com/Setmaster/comfyui-llamacpp.git
cd comfyui-llamacpp && python -m pip install -r requirements.txt

Where people get burned

Empty model dropdown. Models must be in models/LLM/gguf/ (or an LLM root in extra_model_paths.yaml, where a gguf child becomes the root). Restart after changing that config. Projector files must have mmproj in the name to show up as projectors at all.

A vision model ignores your image. Check Server Status to see which projector got resolved. Leave the Start node's projector on (auto) when the right mmproj is installed; if it reports several distinct compatible files, pick one explicitly.

VRAM doesn't drop instantly. Release is verified for the process tree or router residency, not for driver accounting. Give it a few seconds and run the next step before concluding anything's stuck. On Linux the pack also wants kernel 5.4+ for pidfd_open - it checks and refuses with a readable error rather than half-starting.

One caveat on the pack itself: one author, in beta, small user base. What it does have is documentation and a test suite far beyond the usual custom node, which is why I'd reach for it over the older "just call an Ollama endpoint" nodes. Read the docs before filing a bug about behaviour that's deliberate.