llama.cpp Request Budget

The problem it solves

You've wired a 3,000-word style guide into the system prompt, a six-turn history, two reference images and a 4,096-token output ceiling onto a model with an 8k slot context. You queue it. Ninety seconds later you get a response that starts mid-thought and stops mid-sentence, and you spend the next hour blaming your sampler.

The model didn't fail. You asked for more tokens than the slot holds, and something upstream silently did its best. Request Budget counts the complete request - prompt, system prompt, profile wrapping, earlier messages, images, structured-output scaffolding, the whole assembled payload - without generating a single token, and tells you whether it fits.

That's the entire pitch. It's the least glamorous node in the pack and the one that saves the most afternoons.

How the count actually happens

It goes through the same request-preparation path as llama.cpp Generate, so what gets counted is what would have been sent - not a rough character count. Your prompt is rendered with the connected profile, history is appended, media is processed, the model is resolved exactly, and then the payload goes to llama-server's /v1/chat/completions/input_tokens.

Two facts matter and the node is strict about both. input_tokens comes from that count endpoint. context_limit comes only from /props.default_generation_settings.n_ctx - the effective slot context, not the total server context and not the model's training context. Those are three different numbers, and using the wrong one is how people conclude they have room when they don't.

Unknown stays unknown. If the count endpoint isn't available you get null, never a zero. A real zero is preserved as zero, because zero is a legitimate answer. The policy is explicit and local: input_tokens + max_tokens <= context_limit. max_tokens here isn't an output setting - the node generates nothing - it's the output allowance you're reserving. Exactly zero remaining fits; negative overflows.

What you set, what you get

prompt is the only required input, plus everything Generate takes so the count matches reality: system_prompt, messages, profile, connection, server_url, model, image_amount and the image_1…image_10 sockets, structured_output, token_ban, max_tokens, request_timeout.

The one input that's unique to this node is budget_policy, and it has only two values. report - the default - returns fit / overflow / unknown and moves on. enforce fails the node unless both numbers are known and the allowance fits. Counting happens either way; the policy only decides whether a bad result is an observation or a stop.

Three outputs. summary is the human-readable verdict with input count, requested output, effective context and remaining capacity. budget_json is the versioned envelope - payload SHA-256, runtime epoch, elapsed time, release evidence, and deliberately no prompt text or credentials, so it's safe to look at or share. budget is the typed observation for adapters that want to branch on it, which is how you'd wire "if it overflows, regenerate my prompt shorter" without parsing prose.

It is a snapshot, not a permission slip

This is the subtle bit and the pack says it about as plainly as it can: a Request Budget node is a measurement of one payload at one moment. Change the model, the runtime, the prompt, an image, the profile, or the output allowance and the measurement is stale. Generate never accepts another node's green light - its own budget_policy (default off, with report and enforce available) counts its own request again immediately before generation.

So why run it separately at all? Because Generate's enforce fails at the last possible moment, and if you've queued a 32-image caption batch or a long chain, you want the bad news early rather than after the model loads. A budget node upstream of the expensive part is a smoke test. Generate's own policy is the seatbelt.

Nothing in either path truncates, summarises, shortens history or lowers your output allowance. The count is read-only. That's a deliberate choice - a node that silently fixes your request is a node that silently changes your output.

One thing to know before you queue it

Counting is not free. The node needs a running target, takes the runtime's operation lease, and can do real model and media work - loading a router model, processing images, running the tokenizer over the finished payload. In router mode the target has to be loaded first; discovery and label refreshes deliberately don't count anything or autoload a model. So don't put it on a loop and expect a no-op; treat it like a small generation.

It also carries a release_after_generation toggle, which asks the owned runtime for terminal release after counting - handy when the budget check is the last thing before a diffusion step. Check budget_json for the evidence. An attached server the pack didn't start can't be released this way, and terminal release is a process/residency fact, not instant driver memory convergence.

Installing

Part of the pack:

comfy node install comfyui-llamacpp

Or ComfyUI Manager → search "comfyui-llamacpp". The pack's own dependencies are light (aiohttp, numpy, pillow, psutil, requests); the runtime you supply is a llama-server binary, plus a GGUF in ComfyUI/models/LLM/gguf/.

Honest take: if your prompts are short and your context is 32k, you'll never think about this node. The moment you start chaining refinements, feeding long style guides, or attaching images, it earns its slot in the graph.

Where people get burned

Reading remaining as output room. It subtracts both input and the requested allowance. If you set max_tokens to 2048 out of habit but only need 200, the node will tell you you're tighter than you are. Set it to what you actually reserve.

Expecting it to match your chat client. Counts are model- and build-dependent, and multimodal numbers haven't been validated for every model. Compare against actual usage on the model and media type you're using before treating it as gospel.

Measuring a different request than the one you'll send. If the downstream Generate has a different max_tokens or a different model, you counted something else. Keep them identical or the number is fiction.