🎬 Video Prompt Architect (Multimodal)

Video Prompt Architect doesn't generate a single frame. It's a prompt engine: give it a start frame, an end frame, and a one-line idea ("sunset to starry night"), and it hands back a fully-written cinematic transition prompt you can feed straight into a video model. If you've spent an evening writing "slow push-in, warm golden light dissolving into deep blue, stars emerging" by hand and watched LTX or Wan ignore half of it, you get the appeal. Prompt quality is the difference between a usable clip and a 20-second render of mush, and this node outsources the writing to a multimodal LLM that actually sees your frames.

The name is a bit of a lie, in the best way: nothing about it is a ComfyUI model. It's a client for any OpenAI-compatible vision LLM - LM Studio, Ollama, vLLM, a remote server - so it needs no GPU of its own and no model downloads into your ComfyUI install.

How it works

The mechanism is refreshingly simple. The node base64-encodes your start_image (and optionally end_image) as JPEGs, drops them into a chat-completions request alongside a task instruction, and asks the model to reply with only a JSON object: {"en": ..., "tw": ..., "cn": ...}. With two images it tells the model to "analyze the transition… cinematic evolution, camera movement, and lighting changes." With just a start image it asks for an expanded scene description instead - a cheap way to turn a single still into a rich establishing shot.

The three outputs - prompt_en, prompt_zh_tw, prompt_zh_cn - are what you wire onward. prompt_en goes into your video generation node (a Runway/Luma API node, or the text-to-video or image-to-video input on LTX/Wan); the two Chinese outputs are for you to read and sanity-check, or for Chinese-language workflows.

One quirk worth knowing: the seed input isn't a real seed. The code just appends [ID:1234] as noise in the prompt text so repeat runs drift differently. Cute hack, but don't expect deterministic reproducibility - retry with a different seed if a result feels stale.

The inputs that matter

Most fields you can leave alone, but these four decide whether it works at all:

• start_image (required) - the opening frame. It's an IMAGE, so wire it from a Load Image node (or a VHS Video Load).
• end_image (optional) - the closing frame. Leave it disconnected to get single-image scene expansion instead of a transition.
• user_description - your raw idea. The default "A sunset to starry night" is fine as a template; this is where your actual concept goes.
• model_name - must match the model id you actually loaded in LM Studio/Ollama. The default gemma4 is a placeholder that almost certainly doesn't exist on your server; this is the #1 "it 400s immediately" cause.

Behind them: system_role_instruction sets the LLM's persona (the repo ships a SystemPrompt.md full of film-director and MV-director prompts worth stealing), api_url defaults to LM Studio's http://127.0.0.1:1234/v1/chat/completions, and api_key defaults to not-needed for local servers. max_new_tokens (default 2048) and temperature (0.7) behave as you'd expect.

Installing it

Trivial install, because the only real dependencies are requests, numpy, and Pillow - all of which ComfyUI already drags in. Install via ComfyUI Manager (search "Video Prompt Architect") or:

cd ComfyUI/custom_nodes
git clone https://github.com/JetterTW/ComfyUI-Video-Prompt-Architect.git
cd ComfyUI-Video-Prompt-Architect
pip install -r requirements.txt

Restart ComfyUI and you'll find it under VideoProduction → PromptEngine. The real prerequisite isn't ComfyUI-side at all: you need a vision model running somewhere with an OpenAI-compatible endpoint. Text-only LLMs can't read the images and will fail.

Where people get burned

• Wrong model id or a text-only model. The node can't see the images if the LLM can't. Load a real vision model, set model_name to its exact id.
• api_url must end in /chat/completions. The full path, not just the server root.
• Errors come back through the outputs. On failure the node returns the error string in all three outputs, so wire any of them to a Show Text node - it doubles as your error log.
• Slow local models can time out. The request has a 180-second timeout, and there's no image resizing before base64 encoding. Feeding a giant still to a small local VLM is the fastest way to blow that budget; downscale your frames first.
• The request includes a vLLM-style chat_template_kwargs field. Most local servers ignore unknown fields, but if a strict one rejects the payload with a 400, that's the suspect.

It's a niche tool, and at a 0-impressions comfy.icu page it's clearly early-adopter territory - but the idea is sound: stop typing cinematic prompts blind, and let a model that can see your frames write them.