Extensions/ComfyUI-VisionPromptAssistant
ComfyUI Extension

ComfyUI-VisionPromptAssistant

Local multimodal prompt generation and text preview nodes for ComfyUI.

By elgalardi·Created 21 days ago·Updated a day ago· 1
elgalardi/ComfyUI-VisionPromptAssistant
Nodes12
On cloudLocal install
Categorytext, conditioning/minimax
Stars1
Updateda day ago
Readme

Vision Prompt Assistant

A local ComfyUI text-generation node with separate system and user prompts, designed to write reference-aware prompts for MiniMax H3.

The node is intended for full multimodal text encoders supported by ComfyUI, such as Qwen3-VL. It does not call OpenRouter or any other external API.

Recommended local setup:

  • Download qwen3-vl-4b-heretic_int8.safetensors into ComfyUI/models/text_encoders, then select it directly in the node.
  • Start with loader type ltxv, or compare it with stable_diffusion.
  • Connect up to three reference images. Socket names map to MiniMax picture tags in connection order: the first connected image becomes <Picture 1>, the second becomes <Picture 2>, and the third becomes <Picture 3>.

Video and audio sockets are intentionally not included. The recommended Qwen3-VL encoder can inspect images, but it does not reliably understand a complete video or listen to an audio waveform through ComfyUI's text-generation interface. Keeping image inputs only makes the node's behavior predictable.

The loaded encoder is cached and reused while the model, type, and device remain unchanged, so editing prompts does not reread the checkpoint from disk.

max_length is the hard maximum for newly generated tokens. The assistant automatically gives the model a slightly smaller target budget so it has room to finish its final sentence instead of being cut off at the hard limit.

For fuller results, specify the desired approximate length at the end of the user_prompt, choosing a value slightly below max_length. For example, with max_length set to 256, finish the request with: Write about 220 tokens. This gives the vision model an explicit length target and helps prevent an otherwise useful final prompt from ending too early.

The user_prompt and system_prompt boxes accept text directly or a STRING cable using ComfyUI's normal widget-to-input conversion. user_prompt is listed first so it is the preferred STRING route when the assistant is bypassed; the system instruction is not intended as passthrough output.

The MiniMax H3 conditioning encoder is truncated and is not suitable for text generation.

Preview Vision Prompt

Connect the generated string to Preview Vision Prompt to display the full prompt inside the graph without saving it. The text is also passed through as a STRING output for downstream nodes.

Vision Prompt Assistant, Abliteration Vision Prompt, and Preview Vision Prompt are included in this package.

Abliteration Vision Prompt

Abliteration Vision Prompt is a separate API-based alternative that sends the system prompt, user prompt, and up to three images to Abliteration.ai's OpenAI-compatible vision endpoint. It uses the hosted abliterated-model, so it does not load a local vision encoder or consume VRAM needed by the video model.

Create an API key at Abliteration.ai, paste it into the masked api_key field, and connect zero to three images. Connected images are mapped in order to <Picture 1>, <Picture 2>, and <Picture 3>. Images are resized to image_max_dimension and encoded as JPEG before upload; 1024 is the default balance between visual detail, latency, and token cost. When two or three images are connected, the node combines them into one labeled contact sheet before upload. This preserves all references on API backends that process only the first visual block while keeping the MiniMax <Picture n> mapping.

Set thinking to false for faster prompt enhancement. The node returns the generated prompt as STRING plus a second usage_stats string with the token counts reported by the service. A third credits_remaining string reports the credit balance returned after the request, the credits used by that request, and its estimated USD cost when those values are supplied by the API.

The API-key field is visually masked, but—as with many API nodes—the value may still be serialized inside a saved workflow. Remove the key before sharing a workflow JSON or image/video containing embedded workflow metadata, and rotate the key immediately if it is exposed.

Local Whisper Transcribe

Local Whisper Transcribe accepts a native ComfyUI AUDIO value, converts it to mono 16 kHz in memory, and transcribes it locally with faster-whisper. It offers large-v3 for parity with the standalone Captioner and large-v3-turbo for faster experiments, plus automatic language detection, VAD, beam-size and CUDA/CPU compute controls. Outputs include the complete transcript, timestamped segment JSON, detected language and a concise status string.

Models download on first use to ComfyUI/models/faster-whisper. The Whisper backend is deliberately short-lived: after every transcription the node unloads CTranslate2, releases Python references and clears available CUDA cache before MiniMax H3 begins loading. Silent audio is detected before loading the model.

H3 Story Director

H3 Story Director is a multimodal planner for MiniMax H3. It accepts an optional story idea, up to four character or subject images, and an optional source-video IMAGE frame batch from VHS Load Video. It uses an OpenRouter vision model with strict structured output, then validates the response locally before returning a plan_json and the first complete mode-specific prompt.

Director Mode provides three clear production paths:

  • Continuous Story preserves pose, action, camera, location and sound state from one scene into the next.
  • Cinematic Cuts preserves identity and narrative state while resetting camera position, framing, lens, pose and movement after every hard cut.
  • Edit automatically writes still-image generation/edit prompts when no source video is connected. With a VHS IMAGE frame batch connected, it analyzes <Video 1> and automatically infers motion transfer, character replacement or insertion, wardrobe, environment, style, camera/choreography, or object editing from the user's prompt. There is no separate operation selector.

Video analysis samples 10 frames by default, distributed uniformly from 0% to 100% of the VHS IMAGE batch. Each sample is sent as a separate full-detail image instead of being reduced into a contact sheet. The Director reconstructs one chronological progression and must track subject pose, body-part movement, direction, intensity, contact, clothing or visible nudity, visible anatomy, objects, setting, lighting, framing, and camera behavior. Clearly visible adult or explicit content is described directly and objectively without inventing details that are not visible. Observation labels and analysis methodology are forbidden from generation prompts. Analyses shorter than the required detail threshold are rejected before generation. The same source video must be connected to the H3 generation workflow as <Video 1>.

mode_prompt returns the first complete prompt adapted to the chosen mode. source_video_analysis exposes the chronological interpretation for inspection and is empty when Edit has no source video. scene_count is always respected: requesting multiple video-edit scenes produces that many ordered, distinct prompts instead of silently forcing the result to one scene.

For a standalone I2V generation, set scene_count to 1 and connect the scene_prompt output directly to the MiniMax H3 I2V prompt input. This output contains only the complete shared prefix plus the generated scene prompt; it does not include JSON, duration, sampler steps, validation, or usage data. With multi-scene plans it returns the first scene's complete prompt.

The default model is x-ai/grok-4.20, which supports multiple image inputs and strict structured outputs on OpenRouter. The model field remains editable. Reasoning is disabled by default for lower latency and cost and can be enabled for more complex story structures.

Director Profile separates model-specific planning without changing any public output or downstream H3 node. OpenRouter retains the established compact scene schema. Gemma requires a private structured worksheet for every moving-video scene: an explicit-action checklist, duration-aware chronological beats, physical performance, motivated camera, concrete environment and lighting, synchronized sound, dialogue, final state, and a coverage check. The node compiles those private fields into the same MiniMax-ready scene prompt and discards the checks. This keeps OpenRouter stable while allowing future local-model profiles to be added independently.

External / datacenter LLM

H3 Story Director — LLM Model (API) provides the same inputs, multimodal reference handling, validation, plan compiler, and outputs without calling OpenRouter. Its llm_model socket accepts the package's H3 LLM Model (API) output as well as the LLMMODEL output from YALLM's LLM Model (API) or LLM Provider (API) nodes.

H3 LLM Model (API) connects to an OpenAI-compatible datacenter endpoint. Enter either a base URL ending in /v1 or the complete /chat/completions URL, plus the served model ID and optional API key. The key widget is visually masked but may still be serialized in workflow metadata; deployment systems should inject or remove credentials before distributing workflows. YALLM can instead keep connection profiles and credentials in its server-side YAML.

The external Director still performs all final plan validation locally. With a YALLM model that does not expose structured-output parameters, it supplies the exact JSON Schema in the system instruction and rejects malformed responses before any MiniMax render begins.

H3 Ollama Model (Local) is the Ollama-protocol connection for the same external Director. Its default server is http://127.0.0.1:11434, the standard address when Ollama and ComfyUI run on one computer. The address accepts a local, LAN, remote, or hosted server root, an address ending in /api, or the complete /api/chat endpoint; an optional masked API key supports protected services. It uses Ollama's native multimodal chat endpoint so up to four Director images, the exact JSON Schema, seed, temperature, output-token budget, and context length are preserved. thinking=false favors speed; thinking=true lets supported models reason internally before returning the final structured plan. keep_alive=false sends 0 and unloads the VLM before MiniMax begins; keep_alive=true sends -1 and keeps it resident for repeated plans. The currently tested local profile is Gemma with huihui_ai/gemma-4-abliterated:12b. Select the matching Director Profile in H3 Story Director — LLM Model (API); the profile does not select the model itself.

story_idea is optional. Leaving it empty enables Full Creative Control: the Director invents the premise and complete narrative arc from the selected genre, Motion Style, dialogue setting, scene count and duration, additional direction, and any connected reference images. Writing a premise keeps the original guided behavior. New nodes also start with an empty additional_direction field so the example musician story never leaks into an unrelated creative run.

Dialogue is a fixed selector containing No dialogue, ten of the most widely spoken languages by total speakers (English, Mandarin Chinese, Hindi, Spanish, Standard Arabic, French, Bengali, Portuguese, Russian, and Indonesian), plus Japanese for continuity with the original workflow. The synopsis, story bible, JSON plan, and production directions remain in English.

When dialogue is enabled, the Director writes every actual spoken line in quotes, assigns it as (S1):, (S2):, (S3):, or (S4):, and keeps the exchange naturally performable inside the selected duration. It never leaves dialogue for MiniMax to invent. Selecting No dialogue removes spoken dialogue, narration, voice-over, and intelligible background speech.

audio_content independently controls the permitted vocal and musical content. Its first and default option, Auto, infers the appropriate dialogue, singing, instrumental score or intentional ambience from the complete story while keeping audio choices coherent across scenes. Dialogue Only excludes score and singing while retaining ambience and Foley; Dialogue and Music adds a non-vocal score that ducks beneath speech; Singing Music Only removes spoken dialogue and requires exact sung lyrics whenever lyrics are intelligible; and Instrumental Music Only prohibits every spoken or sung voice. The Dialogue language applies to spoken lines and lyrics. When No dialogue is combined with a dialogue mode, speech remains disabled. In singing mode it uses a language explicitly requested by the story or non-lexical vocals.

Genre is an expanded menu covering the original drama, action, thriller, horror, comedy, romance, science fiction, fantasy, documentary, music video, anime, animation and adult categories, plus adventure, crime, detective mystery, film noir, sitcom, slasher, superhero, western, martial arts, heist, espionage, disaster, psychological drama, dark comedy, and musical. Commercial and social formats include advertising, product showcase, fashion, beauty, food, luxury, TikTok/Reels, YouTube, vlog, influencer/UGC, corporate, sports, travel, educational and video-podcast productions. Additional adult formats include OnlyFans-style creator video, glamour/boudoir, pornographic, fetish, sensual romance, erotic comedy, amateur-style, POV, couples, BDSM-themed, consensual fantasy roleplay, intimate art film, and explicit music-video formats. Adult genres require all depicted participants to be clearly consenting adults aged 18 or older and reject sexual treatment of minors or age-ambiguous references.

The first genre option, Auto, is not treated as a literal genre. The Director infers a coherent genre, production format, tone, audience and visual language from the written premise, additional direction, reference images, source-video timeline and Director Mode. Explicit written intent takes precedence over ambiguous visual clues. If the inferred format is adult, the same consenting-adults-only validation remains mandatory.

Motion Style independently controls the global action/camera cadence without changing scene duration. Its first and default option, Auto, infers the best motion intensity and camera language from the prompt, references, source video, genre, and Director Mode. Alongside Normal, Fast, Slow, Time Lapse, Stop Motion, Hyperlapse, Speed Ramp, Minimal, Fluid, and Intense Dynamic motion, it includes Super Fast, Super Slow, Handheld, Gimbal, Steadicam, Locked-Off, Dolly/Tracking, Crane/Drone, Orbit, Whip Pan, and Crash Zoom camera styles. Each selection expands into explicit physical-action and camera guidance for the LLM. More extreme choices include Chaotic Erratic, Frenetic Kinetic, Unhinged Handheld, Surreal Unpredictable, and Pulsing Rhythmic motion; calmer choices include Meditative Calm, Gentle Organic, Dreamlike Floating, Static Tableau, and Slow Observational motion.

The model writes the synopsis, continuity bible, shared reference assignments, and scene prompts. The node—not the model—enforces the requested scene count, duration, and sampler steps. Every connected image must be assigned through its exact <Picture 1>, <Picture 2>, <Picture 3>, or <Picture 4> tag or the plan is rejected. Incomplete JSON, missing scenes, duplicate IDs, and underspecified scene prompts also stop before video generation begins.

Connect plan_json to the plan_json_input socket on MiniMax H3 Contex Loop Plan. After the Director runs, its frontend synchronizes the accepted JSON back into the Plan node's visual scene cards so the story can be reviewed and edited before an expensive render. Keep draft_only enabled for this first run: it blocks downstream video generation while still filling the cards. Once the plan is approved, disconnect the Director cable and queue the workflow; Chain Plan then uses the synchronized local copy without paying for another API request. Start with three scenes at five seconds, then increase the length after validating the workflow.

The OpenRouter API-key field is masked visually but may still be serialized in the workflow. Remove the key before sharing workflow metadata. For hosted installations such as Runpod, leave the field empty and provide the key through the OPENROUTER_API_KEY environment variable instead. A value entered in the node takes priority over the environment variable.