LlamaServe-Doc Loader

Every "run an LLM inside ComfyUI" plan has a quiet moment where you install LM Studio or babysit an Ollama server in the background. LlamaServe-Doc Loader is the answer to that moment: it downloads a native llama-server backend, launches it as a managed subprocess, and hands the running config to its partner node so you never touch a separate app again. This is the loader half of a two-node pack - pick your GGUF and how much of your GPU it gets to eat, and the LlamaServeDocGenerate node does the actual talking.

How it works

The Loader registers a models/LLM folder and gives you a dropdown of every .gguf in it. On the first execute, it downloads the official Windows CUDA 12.4 llama.cpp build from the latest ggml-org/llama.cpp GitHub release - verifying the SHA-256 from the Release API before unpacking - into the pack's own runtime/ folder. After that it reuses the install, so the download cost is a one-time surprise. "Managed" is the operative word: run the graph again with the same settings and it reuses the still-alive server and loaded model; change any Loader parameter and it restarts cleanly. If a foreign process squats on your port, it errors out rather than killing something it doesn't own.

The inputs that matter

• model - dropdown of GGUF files in ComfyUI/models/LLM/. The bundled demo workflow expects Qwen3.6-27B-H3-Prompt-Rewriter-Q4_K_M.gguf; drop it in the same folder.
• gpu_layers - how many layers offload to GPU. Default 47 works on 16GB; -1 lets llama.cpp decide. This is your first VRAM lever - lower it when you get out-of-memory.
• context_length - default 4096, up to 262144. Bigger context eats more VRAM, so raise it only when the job needs it.
• cache_type_k / cache_type_v - KV-cache precision (q8_0 / f16 / q4_0, default q8_0). The cheap second lever when gpu_layers won't drop any lower.
• flash_attention (on / auto / off) - leave it on unless you have a reason.
• port - the localhost port the server listens on, default 8191. Change it if something else owns that port.
• mmproj - optional multimodal projector for vision models. Text-only use selects None; the pack only sends text today, so this is largely future-proofing.

The one output, server_config (LLAMA_SERVER_CONFIG), is not for you to read - it wires straight into Generate, and nothing else consumes it.

Install & gotchas

cd ComfyUI/custom_nodes
git clone https://github.com/DocWorkBox/LlamaServe-Doc.git

Restart ComfyUI and the nodes appear under LlamaServe-Doc in the node list. Or search "LlamaServe-Doc" in ComfyUI Manager. Models go in ComfyUI/models/LLM/. A bundled example workflow (a Qwen prompt-rewriter for joint audio-video generation) loads from Workflow → Browse Templates → LlamaServe-Doc; its text-display node needs ComfyUI-Easy-Use, but deleting that node doesn't affect the Loader/Generate pair.

Read the README's 16GB starting recipe for the demo 27B model: context 4096, gpu_layers 47, flash attention on, q8_0 KV cache. Q4_K_M is the accepted low-VRAM compromise for a reason - the GGUF quality ladder is well-trodden, and Q8 is basically fp16 at half the size if you can fit it.

Two honest gotchas before you commit. First, this pack is Windows + NVIDIA CUDA only - the backend it downloads is the Windows CUDA 12.4 build, and there's no Linux or Mac path. Second, it has zero Python dependencies because the backend is a binary, not a pip package; that's the trade for the "no Ollama" claim. The first run is slow (backend download plus model load), so don't panic at the hang - logs land in the pack's logs/ folder if you want to watch it work.