[llama.cpp] Prefill Decide

Asking a chat model to answer a closed question in free text is a bad idea. It writes you a paragraph, hedges, and then you're regexing "I would say probably yes, although…". Decide is the fix: you hand it a question and a fixed list of candidate answers, and it scores the candidates and returns one.

This is the node for the branch in your graph that has to be a decision, not a caption: is this shot day or night, is the subject wearing a hat, does this take need a re-roll, which of these four prompt variants fits the image. Some workflows call this pattern "system one" - one token, no deliberation.

How it works

The candidates are turned into single-token labels (A, B, C … up to 26) and the model is asked, under a strict output rule, to return exactly one label with no explanation. Two implementations sit behind that:

• On a Native Session it reuses the loaded model and scores the candidate tokens straight off prefill logits - deterministic, no sampling, no seed.
• On a Runtime or Connect Session it goes through the server: /apply-template to build the prompt, /tokenize to verify the labels are single tokens, then a grammar-constrained one-token /completion. The candidate probabilities are renormalised over the candidate set.

Both paths return selected as your original answer string, not "A". probabilities_json is an object mapping each answer to a probability in your input order. Take those numbers as relative ordering, not as calibrated confidence - the pack's own design notes are explicit about that, and they're right. A 0.72 vs 0.28 split does not mean the model is 72% sure.

The question itself arrives on a dedicated socket type, so you feed it from the pack's [llama.cpp] Create Question From Input node, which takes a question string and a connected STRING list for the answers. That's the plumbing detail that confuses people: answer is a ComfyUI list wire, not one long comma-separated string. Validation is 2–26 answers, non-empty, and all unique - duplicates are rejected, not silently deduped.

Optional images / audio / video are appended to the decision context as MTMD content. The question and answers stay text-only; the media is evidence, not a second question.

The inputs that matter

session - from Create Runtime Session, Connect Session, or Create Native Session. Required.

question - the question/answer payload from Create Question From Input.

context - extra text fed in ahead of the question: the caption you generated two nodes ago, a list of style constraints, whatever the decision should depend on. system sits above that. In the current implementation these are separate strings rather than a real system role, so don't expect the "system" half to have special weight.

video_with_audio - set it when the video you're passing carries its own audio track and you want the audio considered.

seed - only affects the runtime/server completion path. Native prefill scoring ignores it, because it's deterministic. This catches people out when their "same seed" comparison across two session types doesn't behave the same.

session_unload - unload the session once the decision is done. Handy at the end of a chain.

Outputs: selected, probabilities_json, metrics, media_diagnostics, and session passed through so you can chain another node or unload.

Install

Manager → search llama multimodal → ComfyUI-llama-multimodal. Or:

cd ComfyUI/custom_nodes
git clone https://github.com/craftingmod/ComfyUI-llama-multimodal.git

ComfyUI 0.19.3+. The pack's requirements.txt installs makoto-decision, and you genuinely need it here - the Native Session scoring path is built on that library and fails without it. The optional multimodal llama-cpp-python fork only matters if you're using Native sessions.

For a Runtime Session you'll also need the llama binary (Settings → llama-multimodal → Download llama.cpp, or your own on PATH); gguf models and their mmproj go in ComfyUI/models/LLM.

Where people get burned

"llama.cpp server did not generate a valid decision label." The candidate list couldn't be turned into distinct single tokens. Long answers, answers starting with the same word, or an unusual tokenizer will do it. Fix by shortening candidates to a few words each - "day" / "night" is better than "it appears to be daytime" / "it appears to be nighttime" anyway.

Endpoints. Connect Session against a server that doesn't expose /apply-template and /tokenize will fail; those are llama.cpp server endpoints, so an Ollama server or a random OpenAI shim is out of scope.

Don't reach for this to rank long free-text answers. It's built for atomically tokenizable choices. If your options run to sentences, use Generate and compare, or reshape the task.

Reasoning models make it worse, not better. Thinking output in front of a one-token answer is exactly the failure the KB describes for local LLMs in a graph - scratch-work bleeding into the result. If your model supports it, turn thinking off for decisions; that's what the Thinking / Reasoning Profile node is for.