arkennemasis Local LLM (GGUF, llama.cpp)

An LLM inside the graph, running on your own GPU, from a GGUF file you dropped in ComfyUI/models/LLM. No Ollama server, no account, no per-call cost, and no filter. That combination is what the local-LLM corner of this ecosystem has been asking for - the standing wish is "use any GGUF LLM directly in a Comfy node without Ollama in the loop," and this node is that wish, done with a twist that matters more than the features.

The twist is process isolation. Each call runs a fresh Python interpreter, answers, and exits. llama.cpp allocates its own CUDA memory outside ComfyUI's model management and doesn't hand it all back when the object is deleted, so a 26B model left resident would sit on 16+ GB while the video models that come after it try to load. Killing the process is how the VRAM actually comes back. The node also calls the pack's release step before it starts, in case a previous stage left video weights occupying the card.

What you feed it

model is a dropdown of every .gguf under ComfyUI/models/LLM. Instruction-tuned models only - the chat template is read from the file itself, and that's the only template guaranteed to match the weights. Pair a model with the wrong template and you get fluent-looking nonsense rather than an error, so don't go reusing a template from somewhere else. Vision projector files (mmproj*) are deliberately skipped, so this is text-in, text-out; no image input.

system and prompt are the two messages, both wired inputs, so you can drive them from other nodes - a Job/Story brief, a scene list, a caption. temperature defaults to 0.7, max_tokens to 4096 (up to 32768), seed to 0.

The optional json_schema is the clever one: wire a schema and decoding is constrained to it, so the answer is always parseable JSON of that shape. That's how you get a 30-scene plan out of a model without it wrapping the answer in "Sure! Here's your JSON:" and markdown fences. The honest cost, from the author's own measurement on a 26B model: about 81 tokens/s free-running versus 7.6 tokens/s under a JSON grammar. Roughly a 10x slowdown. If your caller can validate and re-ask, ask for JSON in the prompt instead and keep the speed.

context_tokens defaults to 16384 and goes up to 131072. Bigger context is more VRAM, and it's the setting people quietly forget when a 4B model won't co-exist with the rest of their graph.

Outputs: text and report, where the report gives you the model basename, tokens generated, elapsed seconds, load time and finish reason. Worth wiring somewhere visible: when finish_reason is length the node also prints that the answer hit max_tokens and is cut off, which is the difference between "the model is dumb" and "you asked for a novel."

Install - including the part the README doesn't say

The pack installs the usual way:

cd ComfyUI/custom_nodes
git clone https://github.com/Hishamahmer/comfyui-arkennemasis
pip install httpx

But requirements.txt only declares httpx, and the LLM worker imports llama_cpp inside the child process - so the node loads fine and then fails on Run if llama-cpp-python isn't there. Install it yourself, built for your CUDA version (the project publishes wheel indexes per CUDA toolkit):

pip install llama-cpp-python --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu124

Then drop a GGUF in place and restart:

mkdir -p ComfyUI/models/LLM
# e.g. ComfyUI/models/LLM/your-model-Q4_K_M.gguf

Quant choice is the usual ladder: Q8 is basically fp16 at half the size, Q5 is the last rung with no visible loss, Q4_K_M is the accepted compromise on a 12 GB card.

Gotchas

• Missing llama_cpp doesn't show up until you run it. The failure reads "the worker died without answering" followed by the child's log tail, which will name the real error. If you see that, it's the dependency, not the model.
• This node is not an output node. A graph that ends here won't execute - wire text into something that is (a save, a preview, another node's input) or nothing happens.
• The answer is cut off at max_tokens. Raise it, or lower your ambition; the report tells you which happened.
• Big context and big models compete with everything else. The process-exit trick means the VRAM comes back after the call, not during it. Budget for the model and whatever else is resident.
• If you'd rather keep an Ollama server running, that's a legitimate choice - the tradeoff is one process versus two, and whether you want the model managed outside ComfyUI.