Qwen2.5-Omni Multimodal Prompt Assistant

Every prompt assistant in ComfyUI is vision-only. You feed it a video and it looks at the frames - the pack's other assistant says so in its own docs, "audio and video soundtracks are ignored." Which is fine right up until the thing you want described is the sound: a beat, a spoken line, a door slam that tells you what the shot is about.

HenjiccOmniPrompt - Qwen2.5-Omni Multimodal Prompt Assistant, Henjicc/Text - is the one that hears it. It runs a local Qwen2.5-Omni over your images, video frames and audio together, and prints text. One required model, one optional, one output.

What it does

It walks your media bundle (max eight assets), labels each one in the prompt (<image1>, <Picture 1>, <audio1> - matching whatever profile the Super Input resolved), and builds the multimodal inputs itself: images EXIF-corrected and thumbnailed to 672px, video sampled at 2 fps for the first analysis_seconds, audio resampled to 16 kHz mono and cut to the same window. Then it loads Qwen2_5OmniForConditionalGeneration from models/LLM/Qwen2.5-Omni-3B (or -7B) in bf16 with SDPA attention and audio output disabled, and generates through the model's thinker path - greedy, no sampling - with a wall-clock deadline checked on every token.

That deadline is timeout_seconds, and it's a real one: the stopping criteria raises once time.monotonic() passes the deadline, so a wedged run dies instead of pinning your GPU for the evening. Before loading it calls free_memory for 16 GB (3B) or 28 GB (7B) and clears its own cache afterwards, which is the polite version of "this node is a houseguest."

The output is one text socket. No re-encoded media, no state round-trip. You wire it into a CLIP text encode or a save-text node and that's the workflow.

The inputs

• task - a fixed list: describe (describe what's there), qwen_edit (rewrite into a Qwen-Image edit prompt), character_card (rewrite into a turnaround-sheet prompt), h3_swap (rewrite into MiniMax H3 role-replacement language). Each one is a hand-written system instruction, which is the sane way to build this: a scoped job drifts less than an open "improve my prompt."
• analysis_seconds - 1 to 15, default 5. The head of the audio and video that gets analysed. Not a trim; it never sees past this.
• max_tokens (64–2048) and timeout_seconds (30–900, default 300). If you hit the token cap mid-sentence the node raises "Output truncated" rather than handing you half a prompt.
• model_name - Qwen2.5-Omni-3B or Qwen2.5-Omni-7B. Only these two, and only from models/LLM/<name>.

Setting up the model

This is the part that stops people. The node will not download anything, and it checks the model is actually complete: model.safetensors.index.json must exist, every shard it lists must be on disk, and so must spk_dict.pt. Miss the speaker dictionary and you get "Omni model download is incomplete," which is at least an honest error.

# in the Python env ComfyUI runs on -- optional dependencies from pyproject.toml
python -m pip install "transformers>=5" soundfile librosa accelerate

# full model to ComfyUI/models/LLM/Qwen2.5-Omni-3B/ (shards + config + spk_dict.pt)
huggingface-cli download Qwen/Qwen2.5-Omni-3B --local-dir ComfyUI/models/LLM/Qwen2.5-Omni-3B

Those four packages are the pack's omni extra; the base pack only needs aiohttp av numpy Pillow. Install the pack itself with Manager (comfyui-henjicc) or a git clone into custom_nodes, then restart.

Where it gets you

CUDA only. The code raises "This test node requires CUDA" on anything else. The author's own comment calls it a test node, and that's the right expectation to set - this is the least-battle-tested corner of the pack, with no English-language community footprint yet, so judge it on source rather than reputation.

16384-token context. Eight assets, 15 seconds of video and a long prompt will overflow it. Reduce assets or seconds; the error names both.

It narrates, it doesn't transcribe. The tasks ask for descriptions and prompt rewrites, not verbatim dialogue. If you need a transcript, that's a different job - and per the audio ecosystem's own honest limits, TTS and song generation are where local tooling is strongest; understanding is thinner.

Trust it like any VLM. Multi-subject attribution is the standing weakness of every captioner here, and an assistant that "improves" your prompt can add things you never asked for. Read the output before it goes into a conditioning slot.