ComfyUI Node

PainterAV2V

Wrong lips on footage you already rendered? PainterAV2V re-syncs them to new audio

By princepainter·Created 7 months ago·Updated 4 days ago· 191
PainterAV2V
  • model
  • model_patch
  • positive
  • negative
  • vae
  • audio_encoder
  • video
  • mask
  • start_image
  • clip_vision_output
  • model
  • positive
  • negative
  • latent
width832
height480
length81
fps25.0
audio_scale1.00

Most lip-sync nodes generate a talking head from scratch. PainterAV2V, the streamlined sibling in the same pack as PainterAI2V, does the opposite: you hand it video you already have plus an audio track, and it re-animates the mouth to match the new words. Same InfiniteTalk lip-sync engine underneath, but as video-to-video instead of image-to-video. If you've got a clip where the character is talking but the lips say something else - or you want to re-voice a scene with a different line - this is the node for the job. "AV2V" is literally the pitch: audio drives the video-to-video pass.

Here's what it isn't, so you don't waste an evening: it is not a text-to-video node, and it won't animate a still portrait by itself. The video input is an IMAGE batch (a rendered sequence), and that's the footage it works on. You feed it the clip from VHS Load Video or an ImageFromBatch, and the node VAE-encodes those frames straight into the latent it hands back. Because the video is already encoded, the sampler refines the existing motion instead of generating blind - that's the whole reason re-lip-sync needs far fewer steps than a from-scratch generation.

Under the hood it's the same trick as its big sibling, minus the dual-model machinery. One Wan model gets patched with InfiniteTalk's outer-sample wrapper and a cross-attention audio patch; audio features from the encoder (which arrive at 50fps) are linearly interpolated to your target fps so the mouth lands on the right frames. An optional mask restricts where the lip-sync applies - handy when only the head needs re-voicing and you don't want artifacts bleeding into the body. Optional start_image and clip_vision_output anchor the generation the way they do in the official flow.

The inputs that matter

  • video - the frames to re-sync. This is the whole point; everything else is support.
  • audio_encoder - the output of ComfyUI's AudioEncoderEncode node. Don't feed it a raw audio file; it wants the encoded features.
  • fps - 1–60, default 25. Match it to how you'll render. It's the smaller range than PainterAI2V's 1–120, because here you're syncing to footage, not inventing a rate.
  • model - a single Wan I2V model (the workflow ships a Wan 2.1 / 2.2 pairing setup, but this node takes one). Paired with model_patch, the InfiniteTalk patch model.
  • audio_scale - lip-sync strength, −10 to 10, default 1. Crank it if the mouth is too subtle, drop it if the audio is fighting the performance.

Outputs are model, positive, negative, and latent - the patched model and conditioning go into your KSampler, the latent is the encoded footage, and you decode the result with VAEDecode. Same length gotcha as every Wan node: keep it at 4n+1 frame counts (81 default) because the VAE compresses time ~4x, or your clip comes back short. And the FP8 note in the source is real: if your model is FP8 the node falls back to FP16 for the audio compute tensors, so don't panic when you see mixed dtypes in the console.

Installing it

Same pack, same install as PainterAI2V - you get both nodes together. ComfyUI Manager (search "ComfyUI-PainterAI2V"), or:

cd ComfyUI/custom_nodes
git clone https://github.com/princepainter/ComfyUI-PainterAI2V

Restart, and you're done: no pip requirements, no bundled weights. The dependency you actually have to satisfy is a ComfyUI new enough to include comfy.ldm.wan.model_multitalk and the audio encoder nodes - if it errors on that import, update ComfyUI before anything else. Models (the Wan I2V UNet, the Wan2_1-InfiniTetalk-Single patch, the wav2vec2 audio encoder, a Wan VAE) come via ComfyUI Manager or the official InfiniteTalk setup; the shipped workflow also wants sibling nodes like PainterAudioCut that live in the author's other pack, so budget a little time to either install those or swap in stock equivalents.

The honest caveat is cost and patience. Re-lip-syncing is faster than from-scratch generation, but it's still an 81-frame-window Wan pass on a big model - expect a 4090-class GPU and minutes-to-hours depending on length. For the workflow where you've already got the footage and just need the mouth fixed, that's a bargain compared to regenerating the whole scene.

Categoryconditioning/video_models

Inputs (15)

NameTypeDefaultDescription
modelMODEL
model_patchMODEL_PATCH
positiveCONDITIONING
negativeCONDITIONING
vaeVAE
widthINT83216–16384
heightINT48016–16384
lengthINT811–16384
fpsFLOAT25.01–60Video frame rate for lip sync alignment
audio_encoderAUDIO_ENCODER_OUTPUT
videoIMAGE
audio_scaleFLOAT1.00-10–10
maskoptMASK
start_imageoptIMAGE
clip_vision_outputoptCLIP_VISION_OUTPUT

Outputs (4)

NameTypeDescription
modelMODEL
positiveCONDITIONING
negativeCONDITIONING
latentLATENT