Audio-to-Latent Modulator 🎧

Audio-reactive video - where the image pulses, shakes, or changes with the music - is the showpiece trick of the ComfyUI video scene, and this is the node that wires the music into the picture. Audio-to-Latent Modulator takes an audio track, measures its amplitude envelope, and turns it into two things: a per-frame FLOAT curve you can drive almost any parameter with, and an optional LATENT batch where each frame's noise is scaled by the audio. The pack's framing nails the positioning: other nodes already produce per-frame values from a frame index - this one produces the same kind of value, but derived from real audio instead.

How it works

It slices the audio into num_frames equal time-slices and measures each slice's RMS (measure = "rms") or peak ("peak") amplitude, normalizing the result to a 0–1 envelope. smoothing (0.3) tempers the curve so it's a flowing wave rather than a spiky mess; gamma (1.0) shapes the response curve - crank it up to make quiet passages count more, or below 1 to emphasize the peaks. seed just controls the random noise if you use the latent output.

The outputs, and how to use them

• value_at_frame - a FLOAT, the envelope value at frame_index. This is the magic socket: wire it into any FLOAT input - denoise strength, LoRA weight, a strength slider you've converted to an input - and that parameter pulses with the audio. For a video of num_frames frames, change frame_index per frame and you get a beat-synced animation.
• envelope_csv - the full per-frame curve as a comma-separated STRING, for scripts or other nodes that want the whole sequence at once.
• audio_scaled_latent - a LATENT batch of shape (num_frames, …) where each frame's noise is scaled by its envelope value. Connect a template LATENT (any correctly-shaped empty latent) to the latent_template input and you get a ready-to-sample batch of audio-driven latent noise. Feed that to a KSampler and you get generated frames whose noise structure follows the music.

That last one is the "proper" audio-reactive path: instead of modulating a parameter per frame, the latent itself is shaped by the audio, so every frame of the video is literally conditioned by the beat.

The honest review

This is the most novel node in the pack's audio set, and it's genuinely the thing people build audio-reactive workflows around. The caveats: the envelope is a blunt instrument - it measures loudness, not musical events, so a track with a flat loudness profile produces a boring curve. And the per-frame wiring pattern (one frame index per render pass) means real audio-reactive video usually involves batching all frames, not a single still. Pair it with Audio Beat Detect for the discrete "where are the hits" view when you need event timing rather than a continuous curve.

Install

ComfyUI Manager → search OmniNodes, or:

cd ComfyUI/custom_nodes
git clone https://github.com/TensorVizion/OmniNodes

Restart ComfyUI; it's under TensorVizion/Audio. Pure NumPy/PyTorch, no extra dependencies.