Jolly Cosmos3 Structured Prompt

Cosmos3-Nano isn't prompted like other models. It was trained against a structured JSON schema with 24+ fields - subjects, lighting, cinematography, timed actions, segments, and a second-by-second temporal_caption - and the model card's own inference path expects you to produce one. Hand-writing that by heart is miserable. JollyCosmos3StructuredPrompt is the shortcut: you type "a Pixar-style red tomato dancing on a disco floor," a local LLM writes the JSON, and the node hands you a ready prompt plus a matching negative.

It's the classic "LLM as a node in the graph" pattern, just aimed at a very demanding format. The node builds a per-mode system prompt containing the full official schema (t2v, i2v, and t2i variants), hardcodes your resolution, aspect ratio, duration, and fps into it so the LLM can't drift, and asks for bare JSON back. On the parsing side it strips markdown fences and bracket-scavenges if the model wraps its answer in prose - because chat models do that, and the node knows it.

What you actually set

• mode - t2v, i2v, or t2i. This picks the schema variant; i2v gets extra guidance to describe the scene "coming alive" from frame 0.
• description - your plain-English idea. That's it; that's the whole job of this node.
• api_url and api_endpoint - the one real setup step. Default is an OpenAI-compatible server at http://127.0.0.1:8890 (llama.cpp territory), or switch api_endpoint to ollama /api/chat for an Ollama server.
• model_name - leave blank and Ollama defaults to qwen3.6-27b; with the OpenAI path you generally need to name your model.
• resolution_preset / aspect_ratio - defaults are the official 480p (832x480) and 16:9. Pick custom to type your own width/height.
• duration_seconds / fps - defaults 7.9s at 24 fps, matching the generation nodes' 189-frame default.

Outputs are prompt and negative_prompt, both plain STRINGs - wire them straight into JollyCosmos3TextToVideo, JollyCosmos3ImageToVideo, or JollyCosmos3TextToImage. The negative output is a fixed JSON blob the pack ships; don't fight it.

One nice safety net: if the LLM call fails or returns garbage, the node doesn't die - it falls back to a minimal JSON that stuffs your description into temporal_caption and carries on. The resolution and duration fields are force-set afterward regardless, so the JSON stays schema-valid even when the LLM is uncooperative.

The optional image input

In i2v mode you can feed an image for a vision-capable LLM to look at. Worth knowing: the node only actually sends the image on the Ollama endpoint path - the OpenAI-compatible path sends text only. So if you want vision, point it at Ollama with a VLM like a Qwen-VL model.

Is it required?

No. The generation nodes accept plain English too, and the pipe handles it - they run the prompt through the Qwen2 chat template either way. But you're leaving a lot on the table: the structured JSON is what the model was trained to read, and temporal_caption is the field that carries the motion description. If you've ever seen a video model ignore half your prompt, this is the fix. It's also worth remembering the pattern's honest ceiling - a small local LLM is a convenience that kills the blank page, not a writer, so keep your description specific and the model will stay closer to it.