H3 Lip Sync Transfer (stable face + driven mouth)

The workaround that shouldn't have to exist, packaged politely

Here's the situation this node exists for. In H3, an image anchor and audio conditioning fight each other: chain your long render on the previous frame and you get one continuous person whose mouth stops following the soundtrack after the first window; unchain it and the lips track beautifully while the subject quietly becomes somebody else at the seam. You can have continuity or performance, not both.

So you render twice. Pass one is the identity pass - the same audio graph with chain_frames on, all the way through. Pass two is the performance pass - identical but unchained, so the mouth actually works and the face drifts. Then you take the mouth motion out of pass two and graft it onto the stable face from pass one.

That's expression transfer, and LivePortrait does it well: it's a video-driven retargeter that warps a driving performance onto a source portrait while leaving the source's identity intact. H3LipSyncTransfer is that three-step recipe collapsed into one node.

What it actually does

You feed it source_images (the chained take - stable face, dead mouth) and driving_images (the unchained take - correct mouth, unstable identity). It writes both batches to temporary mp4s, shells out to your LivePortrait checkout:

python inference.py -s source.mp4 -d driving.mp4 -o out \
  --audio_priority source --animation_region all --driving_multiplier 1.000

then reads the animated result back as frames and returns them with the audio, shifted by audio_offset_frames for the mux. The temp I/O and the subprocess environment are handled for you.

And it works. On the author's 10.33 s / 248-frame two-window test: chained-only scored 0.278 in window 1 and 0.000 in window 2. Through this node, 0.352 and 0.340 - both windows clear the good threshold, the face stays the same person, and the residual offset drops to −2 / +0 frames. Roughly 2–3 minutes for a 10 second shot at 864×480, plus temp disk for the two mp4s.

The inputs

source_images and driving_images are the two renders - same shot, same audio, differing only in chain_frames. It uses the shorter of the two lengths, so don't mix takes.

audio is what comes back shifted; it doesn't drive anything at this stage, the motion is already baked into the frames. fps must match the render.

Then the two LivePortrait controls that actually change the look. animation_region decides what gets transferred - all is what was measured here and moves expression, pose and eyes together. lip moves only the mouth, which is the safer choice for long takes where you don't want the driving pass's head wobble landing on your stable face. driving_multiplier (0.6–1.6) is expression gain: LivePortrait's own knob. If the transferred performance comes out timid, push it to 1.2–1.4; if the face over-articulates, come down.

Optional and both machine-specific: python_exe (an interpreter that can import LivePortrait's deps - torch, insightface, onnxruntime, imageio-ffmpeg, pykalman) and liveportrait_dir (the checkout, working directory for inference.py). The defaults are Windows paths from the author's machine; elsewhere you'll change both.

Outputs: frames, audio, report. Frames into VHS_VideoCombine or CreateVideo, audio into the same node's audio input, report into a text display or just read the console.

Install

The node is part of the pack:

cd ComfyUI/custom_nodes
git clone https://github.com/lebao-arch/comfyui-h3-audio-drive-auto
pip install soundfile   # imported at load time by another module in the pack

Then the part that isn't in the pack: a LivePortrait checkout with weights in it. The author cloned from the gitcode mirror because the github.com remote kept dying with SSL_read: unexpected eof:

git clone --depth 1 https://gitcode.com/gh_mirrors/li/LivePortrait.git

Weights come from ModelScope AI-ModelScope/LivePortrait (base_models/, retargeting_models/, landmark.onnx, plus insightface/models/buffalo_l/), laid out in pretrained_weights/ the way the repo expects. Extra Python deps: pykalman and tyro, both wheel-installable. Two things before you commit: LivePortrait's detection path pulls InsightFace's ONNX models, which are non-commercial even though the library is MIT (Kijai's ComfyUI port swapped in MediaPipe for detection for exactly that reason), and none of it goes into ComfyUI's own environment - it runs in the python_exe you point at.

Troubleshooting

cublasLt64_12.dll is missing / Error 126. The classic. onnxruntime wants CUDA 12's cublasLt, newer torch wheels ship CUDA 13, and ComfyUI's process PATH doesn't include the toolkit. The node prepends the CUDA bin and torch's own lib to the subprocess PATH to dodge it, falling back to torch's dirs when no toolkit is installed - if you still hit it, that's a mismatch between your interpreter and your toolkit, not a graph error.

no inference.py under ... - liveportrait_dir is wrong. It must be the folder that directly contains inference.py and the pretrained_weights tree, not its parent.

Exit code 0 but no video. LivePortrait writes <source>--<driving>.mp4; the node ignores the _concat preview file. If the output folder is empty, read the report/console - the node surfaces the tail of stderr when the run genuinely fails.

Frame count. Standalone LivePortrait dropped 3 frames off the tail of the author's 248-frame test; through this node it kept all 248. Verify in report if you're feeding something frame-exact.

It still scores weak, especially window 2. Transfer can only copy motion that exists in the driving pass, and chained H3 underexposes the performance. If you're getting 0.15-ish, raise driving_multiplier before you touch anything else. If you want genuinely phoneme-accurate lips, skip this node and re-synthesise the mouth with the pack's Wav2Lip node instead - that's a different tool for a different problem.