Nodes/ComfyUI-NKD-Basic-Tools/😺NKD AV Latent
ComfyUI Node

😺NKD AV Latent

Picture and sound in, one masked AV latent out — without a six-node chain

By Nekodificador·Created about a month ago·Updated 3 days ago· 71
😺NKD AV Latent
  • images
  • audio
  • video_vae
  • audio_vae
  • latent_mask
  • audio_mask
  • latent
â—„audio_modekeepâ–º
â—„fps24.00â–º
â—„ramp_ticks0â–º
â—„ramp_out_ticks0â–º
â—„ramp_shapecosineâ–º

Inpainting a video that carries its own soundtrack is where ComfyUI's wiring gets genuinely silly. For a model that samples picture and sound together - MiniMax H3, LTXV - you need VAE Encode for the video, VAE Encode Audio for the sound, a Set Latent Noise Mask on each, and a Concat AV Latent to join them. Five of those six never change. The one that does, the audio mask, has no node of its own, so people bolt a Solid Mask on and get exactly two options: keep everything or redo everything.

😺NKD AV Latent collapses that chain into one node: frames and audio in, one masked latent out, ready for the KSampler. It exists so the part that actually varies - what happens to the soundtrack - stops being buried in the plumbing.

The one widget that decides everything: audio_mode

  • keep - the original soundtrack survives untouched. Usually what you want, and the default.
  • regenerate - all of it is resampled. This is what you were getting with a plain video mask and no audio mask at all.
  • follow mask - resampled only over the stretch of time a mask is on, to the nearest 1/40 s. This is the setting the node exists for: the sound only changes where the picture changes.

Follow that last one down: each audio token takes the strongest pixel of the video frames it spans, so a mask present on a single frame still reaches the sound - important, because the picture side of a video latent is coarser than a frame.

Inputs that matter

  • fps - get this right or picture and sound drift apart. MiniMax H3 runs 24 fps, and the sound is cut or padded to exactly the number of audio frames the model expects for that many pictures. It's the only input the node needs to know how long the soundtrack should be.
  • ramp_ticks / ramp_out_ticks / ramp_shape - the same run-in/run-out ramp controls as 😺NKD Audio Mask, follow mask only. 0 keeps the seam hard; 8 ticks = 200 ms of easing into each regenerated stretch.
  • latent_mask (optional) - what to repaint in the picture. Leave it unconnected and the whole video is generated. Feed it the latent_mask output of 😺NKD Mask Ops (with its VAE and model connected) and the edit lands on the model's own grid instead of being stretched onto it here.
  • audio_mask (optional) - which moments follow mask follows, in place of the picture mask. White regenerates, black keeps. Only its timing is read, so feed it at frame rate. Does nothing on the other two settings.

Output is a single latent that goes straight into the KSampler.

Install and gotchas

Install from ComfyUI Manager (search ComfyUI-NKD-Basic-Tools) or:

cd ComfyUI/custom_nodes
git clone https://github.com/Nekodificador/ComfyUI-NKD-Basic-Tools

Restart after. The pack's real dependencies are just onnxruntime and huggingface_hub; this node needs neither, so it works even if the face nodes are greyed out. Two things to watch: fps is the classic foot-gun (a wrong value desyncs the clip silently), and for H3 specifically, remember the community licence excludes the US, EU, UK and South Korea - the node is happy to run, but the weights aren't legal there. If keep plus a masked picture is all you need, you probably also want the standalone 😺NKD Audio Mask instead of carrying the whole chain in.

Category😺NKD Nodes/Masking

Inputs (11)

NameTypeDefaultDescription
imagesIMAGEThe video, as frames.
audioAUDIOIts soundtrack.
video_vaeVAE—
audio_vaeVAEResampled to this VAE's own rate first, so the audio doesn't have to arrive at it.
audio_modeCOMBOkeepThis is what decides, always. Keep: the original soundtrack survives untouched. Regenerate: all of it is resampled, which is what you get with no audio mask at all. Follow mask: resampled only over the stretch of time a mask is on — audio mask if you connected one, the picture mask otherwise — to the nearest 1/40 s, the rate the sound is masked at.
fpsFLOAT24.001–240Frame rate of the video going in — MiniMax H3 is 24. It is the only thing needed to know how long the soundtrack should be: the sound is cut or padded to exactly the number of audio frames the model expects for this many pictures. Get it wrong and picture and sound drift apart from each other.
ramp_ticksINT00–40Follow mask only. Run-up INTO each regenerated stretch, in audio ticks (1/40 s): the last ticks of the ORIGINAL sound before it rise toward the cut, so the model may rework the very end of the real audio to land on the generated one; the first regenerated tick is already fully new. 0 keeps the seam hard. 8 ticks = 200 ms.
ramp_out_ticksINT00–40Follow mask only. Descent OUT of each regenerated stretch, back into the original sound. Separate from Ramp In because the exit usually wants a hard cut: a descent there lets the model keep 'transitioning' over audio that should simply resume. Leave at 0 unless the return sounds abrupt.
ramp_shapeCOMBOcosineBoth ramps. cosine eases in and out; linear is the control; high band spends the whole ramp between 0.85 and 0.995, the narrow range of mask values the model actually distinguishes.
latent_maskoptMASKWhat to repaint in the picture. Leave it unconnected and the whole video is generated. Feed the latent_mask output of 😺NKD Mask Ops (with its VAE and model connected) and it lands on the model's own grid instead of being stretched onto it here.
audio_maskoptMASKWhich moments Follow mask follows, instead of the picture mask above: white regenerates the sound of that moment, black keeps it. Only its timing is read, so feed one at frame rate — the picture mask is already coarser than a frame. Does nothing on the other two settings.

Outputs (1)

NameTypeDescription
latentLATENT—