AudioX Video to Audio

Video-to-audio is the headliner feature of the AudioX pack, and this is the basic version: you hand it a batch of video frames plus a text description, and it returns generated audio meant to match what's on screen. This is the foley-from-video trick that video model folks have been waiting for - the "I generated a silent Wan clip, now make it sound real" workflow.

Here's the first thing to understand: the video input is IMAGE, not a video file. It's a batch of frames - the same thing you'd get out of a VHS load node or any frame-sequence source. The node feeds those frames in as video conditioning alongside your text prompt (via a video-only conditioning path that also passes an empty audio tensor, because the AudioX model insists on all three conditioners being present), then runs the standard 250-step dpmpp-3m-sde diffusion pass. It generates duration_seconds of audio - the frames steer the sound, the text steers it more.

Inputs that matter

• model - the AUDIOX_MODEL from the AudioX Model Loader. Required, as always.
• video - your frame batch (IMAGE). Load frames with VHS or your usual video-loading node.
• text_prompt - the steering wheel. Default: "Generate realistic audio that matches the visual content and actions in this video". Be concrete: "rain hitting asphalt, distant traffic" beats "ambient sounds".
• steps - 250 default; iterate at 50–100.
• cfg_scale - 7 default (0.1–20). For video work, this controls how hard the model obeys the text on top of the visual signal.
• seed - -1 for random.
• duration_seconds - 1–30, default 10. Note this is the output audio length; the video frames are conditioning, not a duration source.

Output is audio (AUDIO). From there you can go straight to a save node, or use AudioX Video Audio Combiner to get the audio length-matched to your video frames before muxing.

Where this gets you vs. the enhanced version

The basic node is one dial (cfg_scale) for the whole prompt-video balance. If your result is "video says rain, audio says a thunderstorm and it's fighting you," that's a balancing problem the basic node can't solve - the AudioX Enhanced Video to Audio node splits it into separate text/video CFG scales and weights, which is the difference between "good enough" and "actually tunable." Start here to sanity-check that the pipeline works, then graduate.

Install + the usual suspects

Standard pack ritual - clone, install requirements, download the weights:

cd ComfyUI/custom_nodes
git clone https://github.com/lum3on/ComfyUI-StableAudioX.git
cd ComfyUI-StableAudioX
pip install -r requirements.txt

AudioX weights + config.json from HKUSTAudio/AudioX on Hugging Face go in ComfyUI/models/diffusion_models/ (weights renamed AudioX.ckpt). You'll want ffmpeg installed system-wide (sudo apt install ffmpeg libsndfile1-dev build-essential) since any video frame pipeline and the final mux need it.

The honest caveat: local video-to-audio is still early. Community consensus is that MMAudio is the more mature option, and some folks have been outright disappointed by AudioX on hard cases. This model - 16GB+ VRAM territory, tested on a 4090 per the README - does best on distinct, single-source sounds with a good text prompt. If you get mushy ambience, trim the video to fewer, cleaner actions and write a tighter prompt. It's a fun tool, just don't expect it to reliably nail a dialogue scene, because that's out of scope for every model in this space right now.