Local LLM Generate

First, the distinction that trips everyone up. The LLM inside your checkpoint - Qwen3 on Z-Image, T5 on Flux - is a frozen encoder that turns your prompt into conditioning. You don't install it, you can't swap it, and it runs every generation. Local LLM Generate is the other kind: a language model you drop into the graph as a node, running before generation, on your own GPU. It reads text, optionally looks at an image, and writes text back.

So what do you actually do with it? Write dialogue for a video clip. Turn a rough idea into a structured prompt for a model with a fussy format. Read a frame and describe it so you can feed that description onward. Whatever it is, the answer comes out as a string you wire into the next node.

How it works

The node is thin on purpose - it's a client. The heavy lifting lives in a persistent Local LLM Server the pack runs inside ComfyUI's own process through llama-cpp-python. That server is global and survives between runs, so a workflow with five Generate nodes loads the model once. Your node just posts messages to it: an optional system message and your prompt as the user message, using a plain OpenAI-style chat shape.

The name is honest in one respect worth saying out loud: there is no API, no key, and no cloud. The OpenAI-compatible endpoint the pack optionally exposes is served by your machine, for outside clients - nothing leaves unless you turn external access on yourself.

The inputs that matter

Everything under required is a widget you'll touch, and honestly only two of them matter at first.

prompt is the instruction. system_prompt is the standing brief - it defaults to You are a helpful assistant., which is fine for generic jobs and useless for a job that needs a format. If you want "return only the finished prompt, no preamble," this is where that goes.

system_prompt_preset and prompt_preset are dropdowns that load reusable text from ComfyUI/models/LLM/local_LLM_presets/. Type in the text box and the dropdown flips itself to Custom - the author's own description, and the behaviour you want.

seed is the interesting one. It's request-local to this node. The companion Settings node deliberately does not own it, so standard Control After Generate keeps working even when a Settings node owns the model and sampler. Same widget, same trap as every other seed in ComfyUI: it fires after the run unless you switch widget control mode to Before.

Optional: settings (wire a Local LLM Settings node here when the workflow should own the runtime config), image (one still or a batch), and video_frames (an ordered IMAGE batch, sampled evenly by the settings' frame limit). Vision needs a multimodal GGUF plus a matching mmproj projector - a text-only model ignores the socket entirely.

Outputs

Three strings. response is the answer - that's the one you wire into a text-encode node or a preview. thinking is the model's reasoning, separated from the final content when the template provides it; it'll be empty on a non-reasoning model. info is a JSON dump of what actually ran, including which presets were applied - stick it in a text preview when something isn't behaving and you want to know why.

Installing it

ComfyUI Manager, search the pack title, or:

cd ComfyUI/custom_nodes
git clone https://github.com/Solid-Code/ComfyUI-Local-GGUF-LLM

Then the part people skip. The pack's requirements.txt deliberately installs nothing - the file says so in a comment. llama-cpp-python is not in it and never will be, because a generic pip install can replace your working CUDA build with a CPU-only wheel. You install it yourself, in ComfyUI's Python environment:

pip install llama-cpp-python --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu124

Drop a GGUF anywhere under ComfyUI/models/llm/ (subfolders are fine - the selectors recurse), restart ComfyUI, then open LLM in the side menu and start the model. Nodes talk to that running service; if it isn't started, you get a "Local LLM Server is stopped" error rather than a generation.

One honest note on trust: this is a small, single-author, alpha-versioned pack with almost no community footprint yet. A node that loads model weights and touches the network is the exact shape of thing that got weaponized once before in this ecosystem, so read the source before you run it - it's a handful of readable Python files, and the conservative requirements.txt is a decent sign.

When it goes wrong

No models in the dropdown. The folder is ComfyUI/models/llm - lowercase. The pack registers it with ComfyUI as a model folder and adds .gguf to the accepted extensions, so if you see "No GGUF models found," you're either in the wrong path or you never restarted.

It ignores your image. Text-only GGUF. The socket being connected doesn't make the model multimodal.

It won't load and you're out of VRAM. You're budgeting memory for two models at once now. Set Auto Yield to ComfyUI so the LLM releases its context when ComfyUI needs the GPU and rebuilds it on the next request. And remember context_size isn't free - 32768 tokens of KV cache is a second memory budget, with the Q8/Q5 KV quants as your lever.