AudioX Video to Music

This is the sibling of AudioX Video to Audio with a musical default instead of a foley one. Same inputs, same pipeline, same IMAGE frame-batch video conditioning, same 250-step diffusion - the only real differences are the default prompt ("Generate music for the video") and the job you're asking it to do. It's the node you reach for when a silent video needs a score rather than sound effects.

The mechanism is worth spelling out because it sets expectations. The node feeds your video frames in as conditioning alongside the text prompt (using the same video-only conditioning path as Video to Audio), and generates duration_seconds of audio with dpmpp-3m-sde. The video influences mood and pacing - a tense visual will push the audio somewhere tense - but it does not transcribe the visuals into notes. You are not going to get a melody that perfectly accents each cut. You're getting music that feels like it belongs to the footage, which is a different and more achievable job.

Inputs

• model - from the AudioX Model Loader.
• video - the frame batch (IMAGE). Load with VHS or any frame-sequence node.
• text_prompt - your creative direction: "slow ambient piano, melancholic", "tense electronic buildup". The default is deliberately vague; you'll do better.
• steps / cfg_scale / seed / duration_seconds - standard AudioX dials: 250 steps, 7 CFG, -1 seed, 10 seconds.

Output is audio (AUDIO). To actually attach it to video, run it through AudioX Video Audio Combiner (which length-matches audio to your frames) then mux with a video-encode node.

Install & gotchas

The full install lives in the AudioX Model Loader article - the short form:

cd ComfyUI/custom_nodes
git clone https://github.com/lum3on/ComfyUI-StableAudioX.git
cd ComfyUI-StableAudioX
pip install -r requirements.txt

plus the AudioX weights and config.json from HKUSTAudio/AudioX on Hugging Face in ComfyUI/models/diffusion_models/ (weights renamed AudioX.ckpt).

Realistic expectations, because people do get burned here:

• It's a text-conditioned audio finetune, not a composer. Want style/tempo/mood control? The AudioX Enhanced Text to Music node exists exactly for that - it layers musical attributes onto the prompt. This basic node just passes your text through.
• VRAM is a wall. The README's "min 16GB VRAM, tested on a 4090" line is the honest spec. On weaker cards, keep duration_seconds short and steps low or you'll OOM before you hear anything.
• Matching a fast cut video is a stretch. The model generates one continuous 10-second audio pass; it isn't doing per-shot scoring. Give it a video with a consistent mood and it'll do better.

It's a genuinely novel capability for a free ComfyUI node - a silent clip in, a usable demo-score out, no DAW required. Just treat it as a sketchpad for music ideas, not a replacement for your composer, and you'll be happy with it.