AudioX Multi-Modal Generation

Most of the AudioX pack gives you one generation mode per node. This node is the kitchen sink: it accepts text, video, image, and audio inputs at once and generates audio conditioned on whatever combination you plug in. It's the closest thing in the pack to the "anything-to-audio" promise of the AudioX family, and it's the node to reach for when your source material is weird.

Here's the mechanism, because it matters for expectations. All inputs funnel into one conditioning dict for the AudioX diffusion model:

• text_prompt (required) - always part of the conditioning.
• video (optional IMAGE) - frames are prepared as video conditioning.
• image (optional IMAGE) - a single frame gets turned into a short video by repeating it for the generation duration (at the model's default ~8fps), then treated as video conditioning. In other words, image-to-audio is really "static video"-to-audio.
• audio (optional AUDIO) - fed in as audio conditioning, which is how the model does continuation/conditioning on existing sound.
• Any modality you leave unplugged is filled with empty tensors, because the model requires all conditioners to be present. At least one of text/video/image/audio must be provided or it refuses to run.

Inputs that matter

• model - from the AudioX Model Loader.
• text_prompt - default "Generate audio". You'll want to actually describe it.
• steps / cfg_scale / seed / duration_seconds - the standard four (250 / 7 / -1 / 10).
• video / image / audio - the optional conditioning inputs, in any combination.

Output is audio (AUDIO).

What it's actually good for

• Image to audio. A single photo of a rainy street → ambient rain ambience. The static-video trick works better than you'd expect for ambience.
• Audio continuation. Feed an existing clip into audio with a text prompt and get a continuation conditioned on it.
• Text + video hybrids without switching to the dedicated video nodes.

Install & gotchas

Standard pack setup:

cd ComfyUI/custom_nodes
git clone https://github.com/lum3on/ComfyUI-StableAudioX.git
cd ComfyUI-StableAudioX
pip install -r requirements.txt

AudioX weights + config.json from HKUSTAudio/AudioX on Hugging Face in ComfyUI/models/diffusion_models/ (weights renamed AudioX.ckpt).

Real talk on the edges:

• The audio continuation is the least-proven path here. The code wires the tensor into conditioning, but there's no post-processing to guarantee the output continues smoothly from your clip - treat it as experimental.
• Image input is a frame, not a file. Same IMAGE type convention as everywhere: load a picture with a load-image node, not a path.
• VRAM again. More conditioning inputs = more VRAM. The README's ~16GB/4090 note is the honest floor; image + video + audio together will stress it.
• First run slowness from lazy imports and the pack's startup auto-install - let it cook.

If you mostly want one reliable mode, the dedicated nodes are better tuned. But if you want to jam text, a still, a clip, and a reference track into one generation and see what comes out, this is the sandbox node. It's the pack's most ambitious single node, and "ambitious" cuts both ways.