H3 Lip Sync Wav2Lip (phoneme-level mouth)

When "close enough" isn't

Everything else in this pack tries to get H3 to perform your voice while generating. This node gives up on that and does it afterwards: Wav2Lip looks at the audio, decides what the mouth should be doing on each frame, and repaints just the mouth. Every other pixel - face, hair, hands, background - is left alone. Identity is exact, and the lips land on the phonemes instead of the vibe.

That matters because the in-model route has a hard ceiling. H3 conditioning gets you a mouth that performs in time with a voice; it does not phoneme-align, and across a long render it can stop tracking the audio entirely. Expression transfer (LivePortrait) can't help there either, because it only copies motion that already exists in a driving take. Wav2Lip invents the motion from the audio itself.

The numbers make the case. On the same 10.33 s / 248-frame two-window shot where chained H3 scored 0.278 in window 1 and 0.000 in window 2: this node scored 0.292 and 0.359, whole clip 0.228. Window 2 - the one H3 can't drive at all - is the best result of any approach in the pack. About 2.5 minutes for a 10 second 864×480 shot, and no VRAM beyond the model itself, which is 36M parameters.

How it stays out of your way

Wav2Lip's codebase is from 2020 and drags in ancient pinned dependencies - librosa, an old face detector, its own ffmpeg assumptions. Importing that into ComfyUI's process is how you end up with a broken environment.

So the node doesn't. It writes your frames to a temp mp4 and your audio to a temp wav, runs Wav2Lip's inference.py in a subprocess using whichever interpreter you point at (python_exe), reads the result back, and deletes the temp files. ComfyUI never sees Wav2Lip's requirements. This is the right pattern for legacy tooling and I wish more packs did it.

Inputs and outputs

images is any frame batch - H3 output, a loaded video, whatever. audio is the soundtrack. fps has to match the picture.

checkpoint picks between two files and the tooltip is honest about the trade: wav2lip_gan.pth is sharper, wav2lip.pth has marginally tighter sync and a softer mouth. Start with the GAN one; the softness is the more visible flaw.

pads is the crop margin around the detected face, top bottom left right in pixels, default 0 10 0 0 - a little extra chin. If the mouth clips at the edge of the paste region, raise the bottom value. resize_factor downscales before processing: 1 is full resolution, 2 is noticeably faster on big frames at a quality cost. nosmooth disables temporal smoothing of the face box, which you normally want left on.

Optional: audio_offset_frames shifts the returned track for the mux. Note the deliberate quirk - the mouth was already synthesised against the original audio, so this only moves what you hand to CreateVideo/VHS_VideoCombine. Shift the input instead and lips and sound would move together and the knob would do nothing.

Also optional and machine-specific: python_exe (needs to import Wav2Lip's deps: librosa, torch, face_detection), wav2lip_dir (the checkout, containing inference.py and checkpoints/), and ffmpeg_dir, which gets prepended to PATH because Wav2Lip shells out to ffmpeg for audio.

Outputs are frames, audio, report - frames to a video combine, audio to its audio input, and the report tells you whether the length matched or had to be trimmed/repeated.

Install

cd ComfyUI/custom_nodes
git clone https://github.com/lebao-arch/comfyui-h3-audio-drive-auto
pip install soundfile   # this module imports it at load time; the pack declares no deps

Then Wav2Lip itself, outside ComfyUI's env:

git clone https://github.com/Rudrabha/Wav2Lip.git
# weights: wav2lip_gan.pth -> Wav2Lip/checkpoints/
#          s3fd.pth       -> Wav2Lip/face_detection/detection/sfd/

The node checks those exact paths and fails with a message naming the missing file, which is more than most lip-sync wrappers manage. You'll also need ffmpeg on disk for ffmpeg_dir.

The licence is not a footnote. Wav2Lip is trained on LRS2 and its authors forbid commercial use. That's why the node ships with a path setting instead of bundling the weights - the pack is MIT, the model is not. Research and personal projects only, unless you license it commercially (Sync Labs is the usual route). Same class of trap as the LRS2-adjacent research-only checkpoints that keep showing up in this ecosystem: permissive-looking repo, restrictive weights.

Where it goes wrong

Close-ups look soft. Inherent. The mouth is synthesised at 96×96 and pasted back. On a medium shot you won't see it; on a tight close-up you will. The fix is the two-stage trick from the pack's own workflow: Wav2Lip for phoneme timing, then LivePortrait (H3LipSyncTransfer) to repaint that performance onto the crisp source frames - it trades a little mouth amplitude for the original texture, so expect the score to drop a bit and raise driving_multiplier into the 1.2–1.4 range.

The output frame count came back short. Wav2Lip follows the audio, so a soundtrack longer than the picture yields fewer frames. The node trims or repeats the last frame to keep you in lockstep with the source and says which in report. Trim the audio first if you care about the tail.

Missing s3fd.pth. The face detector is a separate download from the checkpoint and it's the one people forget; without it you get a clear FileNotFoundError before any inference.

Longer audio is a design limit, not a bug: cut the voice to the shot before you wire it in, then use H3LipSyncScore to confirm the result actually tracks.