[llama.cpp] Create Runtime Session

This is the node that answers the standing complaint about local LLMs in ComfyUI: "would be nice if we could use any GGUF model without Ollama in the loop." You pick a .gguf off your disk, optionally pick its multimodal projector, and the node launches a workflow-owned llama-server behind a supervised subprocess on an ephemeral loopback port, waits for it to be ready, and hands you a session handle.

Nothing else is running. No external server to start, no port to remember, no Ollama install. The KB flagged in-graph GGUF loading as the pattern the community wanted for years and was quietly getting - this node is one of the ones delivering it.

How it works

Under the hood the node builds the server command, generates a random API key for that server (secrets.token_urlsafe(32), so the session owns its own key, not you), spawns the process, polls until it answers, and then asks the server what model it loaded - it insists on exactly one. That reported name becomes the session's model ID. --parallel 1 and --models-max 1 are hardcoded: one request at a time, one model at a time, which is what you want when the card is also holding a diffusion model.

Keep it alive and it stays warm for the rest of the graph; the pack also tracks sessions and closes them at prompt end and on process exit, so a crash doesn't leave an orphaned server chewing VRAM. To end it deliberately, wire Unload Session.

If you connect custom_chat_template, the prompt string is written to a temp .jinja file and passed as --jinja --chat-template-file, overriding the template baked into the GGUF, and the temp file is cleaned up when the session closes. Nice touch. It's also the reason the input is force_input - it wants a real string, not a widget you type into.

The inputs that matter

model_path - a combo of GGUF files found in ComfyUI/models/LLM (the pack also honours extra_model_paths.yaml, so a shared LLM folder works). If it says [no GGUF models found], that's the folder being empty, not a bug.

mmproj_path - the matching multimodal projector for vision or audio. Leave it as [none] for a text-only model; a text-only request won't even resolve the projector, which saves you load time. Model and mmproj have to be a matched pair from the same release, or you get vision that silently isn't there.

n_ctx - context window, default 8192. Big images and long captions eat this fast; video eats it faster. This is the number that turns "why did it forget the beginning of the prompt" into a fixable setting.

verbose - dumps server chatter into the ComfyUI console. Turn it on once when something's failing, then leave it off; it's noisy.

The optional typed inputs - model_profile, hardware_profile, reasoning, speculative, prefill_profile - are all "connect a small config node instead of fiddling here". Which handler, which GPU layers, whether thinking is on, image-token budgets, draft-model speculation. Start without any of them; the defaults are sane.

Output is a single session, which feeds Generate, Generate (Media Sequential), Decide, Decide (Media Sequential) and Unload Session.

Install

Manager → search llama multimodal → install ComfyUI-llama-multimodal. Or:

cd ComfyUI/custom_nodes
git clone https://github.com/craftingmod/ComfyUI-llama-multimodal.git

ComfyUI 0.19.3+ required. Then get the runtime: if llama isn't on your PATH, go to Settings → llama-multimodal → Download llama.cpp and let the pack fetch a build for your OS and backend (Linux/Windows CPU, CUDA or ROCm; macOS arm64 CPU). It pins a specific release rather than tracking latest, which is the boring correct choice.

Models go in ComfyUI/models/LLM, model and mmproj side by side:

ComfyUI/models/LLM/gemma-4-E4B-it-Q4_K_M.gguf
ComfyUI/models/LLM/mmproj-gemma-4-E4B-it-F16.gguf

Where people get burned

"Could not find a 'llama' executable on PATH" is the first-run error nine times out of ten, and it means the Settings download hasn't happened. Grab it, restart ComfyUI.

GGUF is not magic VRAM. The quant ladder from the GGUF panel still applies: Q8 is effectively fp16 at half the size and is often faster than Q3/Q4 because of dequantization overhead, Q5 is the last stop before you can see it, Q4_K_M is the accepted compromise for 12GB cards. You're about to run this model alongside your sampler, so pick the biggest quant that leaves room for the diffusion model - and if the session and the sampler fight for the last gigabyte, that's the symptom to expect, not a crash with a clean message.

Two models, one card. If you started the session for a 12B model and then queue a Flux-class workflow, expect swapping, not parallelism. Unload Session exists for exactly this ordering problem.

Model/handler mismatch. Some families need a specific chat handler to see images properly, and the handler is decided at session creation. If vision output looks like the model never saw the picture, look at the Model Profile article before you blame the quant.