JR MiniMax H3 Audio Driven Latent Builder

MiniMax H3 is one of the first open video models that generates audio with the picture - dialogue, foley, room tone, all in one pass. Sometimes that's magic. Other times you already have the soundtrack, and you want H3 to move the pixels to match your audio instead of inventing its own. That's the job of this node: it swaps the audio branch of an official H3 joint latent for an externally encoded one, then locks it so the sampler generates video around it while the audio stays put.

What it is (and isn't)

JR_H3_AudioDrivenLatentBuilder is a surgical replacement node, not a media processor. It does not read audio files, it does not run the MiniMax H3 Audio VAE, and it doesn't decode or mux anything. It takes two latents in and gives you one latent plus a status string out. The pack's own framing nails it: "Video: GENERATE, Audio: LOCKED." If you've ever wished a video model would lip-sync to a real take or follow a beat you actually own, this is the node that makes that workflow possible.

The two inputs that matter

• av_latent - the official MiniMax H3 joint video/audio latent, straight out of JR_H3_DirectedVideoConditioning. Video lives in a [B,24,T,H,W] stream, audio in [B,32,2,T].
• audio_drive_latent - your replacement audio, encoded with the correct MiniMax H3 Audio VAE. The normal chain is Load Audio → VAE Encode (H3 Audio VAE) → this input.

Outputs are audio_driven_av_latent and a status string that actually earns its keep: it reports the time fit (trimmed, padded, or unchanged), batch/device/dtype fit, and the drive mode, so when something's off you'll know exactly where.

How the mechanism works

The node validates both streams hard before doing anything - wrong shape, batch, device, dtype, or any NaN/Inf and it refuses rather than guessing. Then it fits your drive audio to the template audio already inside the AV latent: longer gets trimmed from the tail, shorter gets zero-padded, a batch-1 drive gets expanded to match. No interpolation, no looping, no synthesizing a new tail - if your audio runs short, the audio branch just goes quiet at the end.

The clever part is the masks. The video noise mask is preserved (or generated as ones_like(video) if the incoming latent has none), but the audio noise mask is forced to zeros_like(audio). During sampling, a zero mask means "keep this exactly" - so the audio is locked and the video is denoised/generated around it. That's the whole trick, and it's why the README says audio-driven, not audio-generated.

Load Audio -> VAE Encode Audio (MiniMax H3 Audio VAE) -> audio_drive_latent ─┐
JR H3 Directed Video Conditioning -> av_latent ------------------------------├-> JR_H3_AudioDrivenLatentBuilder -> KSampler

Installing it

This node ships in the JR MiniMax H3 pack (Goldlionren/ComfyUI_JR_MiniMaxH3Node). Easiest route: ComfyUI Manager → search "JR MiniMax H3" → install. Manual route:

cd ComfyUI/custom_nodes
git clone https://github.com/Goldlionren/ComfyUI_JR_MiniMaxH3Node.git
<ComfyUI-Python> -m pip install -r .\ComfyUI_JR_MiniMaxH3Node\requirements.txt

Use the same Python that runs ComfyUI - the README hammers this, because portable installs and launchers hide their own Python and system Python won't pick up your packages. The only hard dependency beyond what ComfyUI already ships is imageio-ffmpeg>=0.5. Restart ComfyUI after installing, and note the pack needs a current ComfyUI with native H3 support (ComfyUI shipped day-zero comfy_extras.nodes_minimax_h3). The H3 weights themselves are big (~42 GB) and carry the geofenced MiniMax H3 Community License - worth a glance if you're in an excluded territory.

Troubleshooting

Most errors here are validation errors and they read like the spec, because they are the spec: "Audio Drive latent must have shape [B,32,2,T]", batch/device/dtype mismatches, "contains NaN or Inf". All of them mean you fed it a latent the H3 AV layout can't use - usually an audio latent from the wrong VAE or an av_latent that didn't come from Directed Video Conditioning. Start with the status output, then check the shapes. The node never silently casts or re-encodes, and that strictness is the reason it never corrupts a latent you're about to spend minutes sampling.