LlamaServe-Doc Generate

The point of LlamaServe-Doc Generate is that you never leave the graph to talk to a real language model. Wire in the config from its sibling LlamaServeDocLoader node, type a prompt, and it streams text back from a GGUF model running on your own GPU - no Ollama server to keep alive, no LM Studio install, no API key, no per-token bill. This is the text half of the classic prompt-enhancer / captioner pattern: an LLM reads your rough idea, rewrites it into a structured, model-appropriate prompt, and the result stays inside ComfyUI. It answers the standing community want of running GGUF models "directly in comfy" instead of through a separate server process.

How it works

This pack is two nodes. The Loader picks the model and boots a native llama-server (the llama.cpp binary) as a managed subprocess; Generate is what actually talks to it. The Loader's server_config output arrives here as a custom LLAMA_SERVER_CONFIG type, and Generate sends it an OpenAI-compatible streaming chat request over localhost (port 8191 by default). Streaming means the first tokens appear while the rest are still being generated - no dead-air wait.

The genuinely nice bit is server reuse. Keep stop_server_after_generate off and the model stays loaded between runs: queue the workflow ten times and only the first pays the load cost, which is the difference between waiting a minute for a 27B model to read in and answering on the first token. If you change anything on the Loader, the node notices the config changed and restarts the server fresh.

The inputs that matter

Most of the panel is standard llama.cpp sampling - max_tokens (default 1024), temperature, top_k, top_p, min_p, repeat_penalty, seed - with sane defaults you can leave alone. Three inputs actually decide your experience:

• system_prompt / user_prompt - the chat messages. In the bundled example workflow the system prompt carries the whole job ("rewrite this into a structured multimodal description with shots, timestamps and a soundscape") and the user prompt is your one-line idea. Keep your instructions in system, your content in user.
• reasoning (off / auto / on) - the one that surprises people. On Qwen-class models with a thinking mode, off sends enable_thinking: false so you get clean answer text instead of the model's deliberation bleeding into your prompt. That's not a corner case: reasoning models leaking scratch-work into conditioning is exactly the failure the community hits with these rewriter workflows. Leave it off unless you actually want to watch it think.
• stop_server_after_generate - false (default) keeps the server and model resident for fast repeats; true tears the server down and frees its VRAM after the run, whether it succeeded or errored. Flip it on when you're done generating, or whenever the same card must also run the diffusion sampler.

Outputs

• text - the generated assistant text as a plain STRING. Wire it to any text display node (the example uses easy showAnything from ComfyUI-Easy-Use, which is optional), a text file saver, or a prompt node that accepts a string.
• performance_json - llama.cpp timing data: tokens/sec, prompt tokens/sec, full timings, the server PID, and whether the server was kept running. This is your tuning instrument; if you're deciding how many gpu_layers you can afford, watch tokens/sec here.

Install & gotchas

Install once for the whole pack:

cd ComfyUI/custom_nodes
git clone https://github.com/DocWorkBox/LlamaServe-Doc.git

…then restart ComfyUI. Or search "LlamaServe-Doc" in ComfyUI Manager. Models live in ComfyUI/models/LLM/ as .gguf files. Full install and VRAM notes live on the Loader page; the short version is that the first run downloads a llama.cpp backend (~a few hundred MB, SHA-256 verified) and this pack is Windows + NVIDIA CUDA only.

Where people get burned: interrupting generation does not stop a persistent server, so the model keeps its VRAM after you hit Cancel - enable stop_server_after_generate or restart ComfyUI to reclaim it. And because this is an output node, it re-executes every queue regardless of unchanged inputs - that's by design, since the model has memory between runs. If a port error shows up, another app owns it; pick a different port on the Loader rather than hunting the process.