🦙 Llama.cpp Router Client (Chat + Vision + Audio + Video + Stream)

The name oversells it a bit. LlamaCppRouterClient doesn't run any GGUF model and it doesn't call any cloud API - it's a tidy HTTP client that talks to a llama.cpp server you already have running on your own machine. If you've been shoving llama.cpp into ComfyUI with Bash nodes or one-off scripts, this is the version that just works: no key, no remote, everything stays local.

Why would you want an LLM inside a diffusion graph? Prompt expansion is the classic - feed in "neon cyberpunk street, rain" and get back a long, detailed prompt, or a structured JSON description that parameterizes the rest of your workflow. People are doing exactly this with local servers, from simple enhancers to JSON layouts feeding Kijai's nodes, and it's one of the fastest-growing ComfyUI niches through 2025-26. This node is the cleanest way to plug into it: it takes a prompt, an optional image, even audio or video paths, and returns the model's answer.

How it works

The mechanism is refreshingly boring, which is a compliment. The node builds an OpenAI-style messages payload and POSTs it to {server_url}/v1/chat/completions on your llama.cpp server. Streaming (SSE) is on by default, so tokens accumulate in real time with a progress bar on the node. Along the way it does the housekeeping that makes local LLMs tolerable:

• Separates thinking from the answer - it picks up reasoning_content / reasoning / thoughts fields from reasoning models, and also strips <think>/<thinking> tags if the model wrote them into the text itself.
• Post-processes text, defaulting to fix_mojibake - the fix for the classic Here’s → Here's garbage that older GGUF tokenizers love to emit.
• In json_mode, it forces response_format: json_object and extracts the JSON for you.
• Multimodal input: an IMAGE tensor gets converted to a JPEG base64 data URL (first frame used), and audio_path / video_path files are embedded as input_audio / input_video content parts for llama.cpp's newer multimodal models.

The inputs that matter

Mostly you set four things and ignore the rest:

• server_url - must match the port your llama-server is listening on. The node defaults to http://127.0.0.1:8888, which is not llama.cpp's stock 8080 or the README's 8082 example. If you can't connect, this is the first thing to check.
• model - set it via the Fetch Models button on the node, which queries the server and pops a picker showing loaded status and modalities. (The sister LlamaCppModelSelector node exists to share one model name across several clients.)
• prompt and system_prompt - the conversation, plus an optional system instruction.
• stream - defaults on for live token output; flip off if you want a single response.

Worth knowing: json_mode (with json_schema_hint appended to the system prompt) when you're feeding structured output into other nodes; temperature (default 0.8), max_tokens (0 = server default), and seed (-1 = random); text_postprocess to tune the cleanup; and unload_after_generate, which calls /models/unload after each run - handy when the LLM and your diffusion model are fighting over VRAM. Optional image, audio_path, video_path, api_key (Bearer token for servers started with --api-key), and stop_custom round it out.

Outputs

Five STRING outputs, all of which wire into other string-accepting nodes or a Show Text node: response (clean answer), thinking (reasoning, empty if none), json (parsed JSON when json_mode is on), raw (the full server response as a JSON string), and model_used. The node is marked as an output node, so it won't get pruned from your saved workflow.

Install

ComfyUI Manager (search "Simple LlamaCPP Client") or:

cd ComfyUI/custom_nodes
git clone https://github.com/ai-joe-git/ComfyUI-Simple-LlamaCPP-Client.git

Restart ComfyUI. Dependencies are just requests and pillow - nothing exotic. The heavy lifting is server-side: you need a GGUF model and, for vision or audio, the matching mmproj file:

llama-server.exe ^
  -m Ministral-3-8B-Instruct.gguf ^
  --host 127.0.0.1 ^
  --port 8082 ^
  --mmproj mmproj.gguf ^
  -c 8192

Troubleshooting

Errors come back as a [LlamaCpp ERROR] ... string in the response output, not a ComfyUI crash - read it, it tells you what failed. The usual suspects, in order: server not running or wrong port in server_url; model name not matching what the server advertises (use Fetch Models, don't guess); and multimodal inputs doing nothing because the server was started without --mmproj - the GGUF alone can't see or hear, the projector file is what makes vision/audio work. And if the answer text looks like mojibake anyway, switch text_postprocess to none and check your server settings rather than fighting the node.