Nodes/Multimodal — LLM (Local-first: GGUF + Transformers)/Multimodal — Context schema builder (template → prompt)
ComfyUI Node

Multimodal — Context schema builder (template → prompt)

A prompt contract so your LLM stops improvising its JSON

By theonetwoone·Created 4 months ago·Updated 23 days ago· 1
Multimodal — Context schema builder (template → prompt)
    • system_prompt_addition
    • prompt_wrapped
    schema_json_template{ "answer": "", "should_update_context": false, "context": "" }
    context_open_tag<context>
    context_close_tag</context>
    base_instructionsReturn ONLY JSON matching the template. If context needs updating, write the FULL updated context inside <context>...</context> (and also fill the context field).
    user_prompt

    Ask a local LLM for structured output and you'll get structured output roughly 80% of the time, and complete nonsense the rest. The classic fix is to stop asking politely and start contracting: pin down the exact JSON template, the exact tags, and the exact rules, all in the system prompt. That's precisely what this node does - it turns a JSON schema template and a few instructions into a formatted "OUTPUT CONTRACT (strict)" block you feed into the LLM node's system_prompt, plus a user prompt wrapped with "follow the contract exactly."

    It's a string-builder, not a grammar-constraint engine. There's no JSON-mode decoding and no token-level forcing here - the contract is prose-level discipline that small quantized models actually respond to remarkably well, but it's still a request. If your model won't obey even a pinned contract, the fix is a stronger model or a smaller max_new_tokens, not this node.

    Inputs

    Four of them, and the defaults are sane:

    • schema_json_template - the JSON you want back. Default is {"answer": "", "should_update_context": false, "context": ""}; edit it to match your own fields.
    • context_open_tag / context_close_tag - default <context> / </context>. These define the delimiters the model should wrap updated context in, and they must match what you use on the parse side.
    • base_instructions - the rule text: return ONLY JSON, write the FULL updated context between the tags, etc. This is your natural-language contract.
    • user_prompt - the actual job for the model (e.g. "look at this image and update the character sheet").

    Outputs, and the wiring

    Two strings, and their names tell you where they go:

    • system_prompt_additionsystem_prompt on the MultimodalLLMNode (append to your base system prompt).
    • prompt_wrapped → the prompt input on the MultimodalLLMNode.

    Then the round trip completes with its sibling node, ContextSchemaParser: the LLM's reply comes back, and the parser pulls whatever's between <context>…</context> out as the updated context, feeding it back into your next run's extra_context. Builder and parser are two halves of a stateful loop - a way to give a stateless local model a working memory of "the character sheet so far," which is the closest this pack gets to a chat-with-memory experience.

    Honest expectations: for a closed-loop character-consistency or editing workflow, this is the difference between the model remembering and the model drifting into oblivion. But it's also fiddly - you're debugging prompt contracts, which is its own art. Start with the default template, get one field flowing, then grow the schema. Contract-first prompting is the single highest-leverage habit for local LLM work, and this node packages that habit into something you can drag onto a canvas.

    Categoryllm/multimodal

    Inputs (5)

    NameTypeDefaultDescription
    schema_json_templateSTRING{ "answer": "", "should_update_context": false, "context": "" }
    context_open_tagSTRING<context>
    context_close_tagSTRING</context>
    base_instructionsSTRINGReturn ONLY JSON matching the template. If context needs updating, write the FULL updated context inside <context>...</context> (and also fill the context field).
    user_promptSTRING

    Outputs (2)

    NameTypeDescription
    system_prompt_additionSTRING
    prompt_wrappedSTRING