MiniMax-H3 Prompt Rewriter Omni (sees and hears)

MiniMax H3 doesn't take a normal prompt. It wants a shot-by-shot description, a diegetic soundscape, a score, speaker IDs, cut timestamps - a whole production brief. You can write that by hand for a 10-second clip; you won't enjoy it, and the model's quality is directly downstream of how well you follow its format. This node is the shortcut: it's the "sees and hears" variant of pytraveler's prompt rewriter pack, running LightX2V's third LoRA on Qwen2.5-Omni-7B. A rough idea goes in one end, the structured H3 description comes out the other, and it never calls an API or asks for a key - it all happens on your own GPU.

What makes it the one to reach for among the pack's rewriters is the references. The 27B rewriter has to be told what a reference image contains in words; the 8B one can look at frames. The Omni rewriter is the only one that hears. Connect the actual picture, clip, or audio file - one growing socket accepts IMAGE, VIDEO, or AUDIO, and there's no wrong slot to plug into - and the model reads the asset itself. It's the only one that covers Ref2AV, the full-reference task, which answers with six fields instead of three: subject_definitions, summary, retention_analysis, detailed_description, overall_soundscape, and non_diegetic_music. The rewritten_prompt output always carries the whole answer, so you only have to wire one thing downstream.

How it works

Underneath, the adapter is LightX2V's LoRA trained on Qwen2.5-Omni-7B, the same model the pack's captioners already use. The task dropdown switches between five modes: T2AV (text alone), I2AV and L2AV (one picture as first or last frame), FL2AV (both frames), and Ref2AV. Only Ref2AV takes clips and sound; connect audio to a frame task and it's refused by name before anything loads, not silently dropped.

The strip of coloured squares under the inputs is the ordering - and the ordering is the meaning. Each connected reference gets a square (blue = picture, green = clip, purple = sound) showing what it'll be called. Drag the second picture to the front and it becomes <Picture 1>. That's why order is load-bearing in Ref2AV: Picture 1 and Picture 2 are not interchangeable, and this is where the ambiguity dies. The state lives in the reference_layout widget, so the arrangement travels with the saved workflow and through the API.

Two details worth knowing before you run it. duration snaps: H3 generates on a 17n+5 frame grid at 24 fps, so asking for 10 seconds actually gets 243 frames (10.13 s), and that number is what gets written into the prompt and quoted back in the alignment line. And max_frames controls how many frames get sampled from a clip, spread evenly - each frame is its own picture to the model, so a 30-second clip at full rate would overflow both the context and your patience.

Installing and running it

Install via ComfyUI Manager (search "MiniMax-H3-Prompt-Rewriter") or:

cd ComfyUI/custom_nodes
git clone https://github.com/pytraveler/MiniMax-H3-Prompt-Rewriter-ComfyUI

Then restart. The safetensors route needs transformers, peft, and accelerate installed into ComfyUI's Python - that's what requirements.txt declares. The GGUF route needs nothing pip-installed at all; the first reference task fetches the official llama.cpp binaries (34 MB, or ~511 MB for CUDA) because those tasks run through llama-mtmd-cli in a subprocess. Smallest realistic setup is a Q4_K_M base plus projector and a Q8_0 adapter: ~6.2 GB download, ~9 GB VRAM. The model dropdown auto-lists Qwen2.5-Omni pairs already in your model folders, so if you already have one, nothing downloads.

The traps

• Quantization buys VRAM, not speed. Measured on this adapter, bfloat16 is fastest (~18.6 tok/s) but needs ~20 GB; nf4 is ~9 GB at four-fifths the speed; int8 is the worst of both worlds - slower than nf4 and bigger. So take the largest quantization your card holds, and on the GGUF route don't worry: llama.cpp's quantized kernels actually are fast.
• A safetensors base shows pictures only. ComfyUI's in-process path has no way to hand the model a sound, so clips and audio are refused there. Pick a GGUF base if you want the "hears" part of "sees and hears".
• keep_model_loaded does nothing on reference tasks - those run in a fresh subprocess that takes the model with it when it exits. The node tells you which happened.
• Near-misses are labelled, not hidden: a Qwen2.5-Omni-3B shows (wrong size for the adapter) and a Qwen2.5-VL-7B shows (vision only, not an Omni build) - its projector has no audio encoder.

This node needs a recent ComfyUI (its growing inputs use the v3 node API), so on an old install it may not register while the rest of the pack does. One more thing that isn't this pack's fault: H3's community licence excludes the US, EU, UK and Korea, so check where you stand on the video model before building a workflow around it. And as with any rewriter, review the output before generating - the model can add details you never stated.