H3 Lip Sync Score (offline)

The check nobody runs until the third render

Audio-driven video has a failure mode that looks like success: the clip renders, the face looks great, the mouth moves - and then you watch it with sound and realise the lips are performing something else entirely. That's what you get when a soundtrack gets swapped after generation, or when the audio never actually made it into the latent.

H3LipSyncScore answers that in one node. The name is honest about one thing: it's offline. No API call, no key, no lip-sync model to download, no network. It's correlation arithmetic on the frames you already have plus the audio you think drove them.

What it measures

Two curves, cross-correlated.

The audio side is simple: RMS energy per frame, resampled to your fps. The picture side is per-frame absolute difference plus vertical-edge energy inside a mouth region. Then it sweeps lags in both directions looking for the offset where those two curves line up best, and returns a normalised peak clipped to 0..1.

The ROI choice matters more than it sounds. roi_mode=auto picks the most temporally active block in the frame using motion alone - deliberately without looking at the audio. That's what keeps the score honest: the region gets chosen on "things move here", never on "things move here with the sound". center is a fixed box, useful as a cross-check. face uses an OpenCV Haar cascade and falls back to auto when a face isn't found, which happens more than you'd like on stylised renders.

Measured numbers, from the author's own runs: a matching track on the reference render scores 0.27 offline and 0.35 through the live graph. A different voice against the same video: 0.00. Silence: 0.00, short-circuited before any correlation happens. Synthetic delays of 3, 5 and 10 frames are all reported exactly.

The thresholds live in the report, not in a magic table: 0.18 and up is "good - the mouth is being driven by this audio", 0.10–0.18 is "weak, check by eye", below 0.10 is "mismatch - this audio very likely did NOT drive the video". Note that 0.35 counts as a strong result here. Lip-sync correlation is a low-ceiling measurement; don't read 0.35 as a C grade.

Inputs and outputs

You set images (decoded frames - feed the batch straight from VAEDecode, VHS_LoadVideoPath or H3AudioDriveLong.frames) and audio. Then fps, which is the knob people get wrong: it must match the actual frame rate of the picture, or the envelope is stretched against the wrong timebase and every offset you get back is nonsense. Default 24 is right for H3 output.

max_lag_frames (default 8) bounds the sweep - and on short clips the node caps it further to roughly an eighth of the frame count, because a wide sweep on a five-second clip has no statistical power and the correlation curve goes flat at the edges. plot toggles the visual.

Four outputs: score (FLOAT), offset_frames (INT, positive means the mouth lags the audio), report (STRING - verdict, confidence, ROI source and an explicit suggestion of how many frames to shift the audio), and plot (IMAGE). Wire the plot into SaveImage / PreviewImage; the score and offset can't go into a display node, so the plot is how you actually see the number during a run. The offset is what you feed back into audio_offset_frames on the driving nodes - that's the whole calibration loop.

Install

It's part of the same five-node pack, so:

cd ComfyUI/custom_nodes
git clone https://github.com/lebao-arch/comfyui-h3-audio-drive-auto
pip install soundfile   # another module in this pack imports it at load time

Restart ComfyUI. No models, no checkpoints, nothing to download for this node specifically. It uses numpy and Pillow, both of which ComfyUI already has, and OpenCV only if you ask for roi_mode=face.

Where it's misleading

Below eight frames it gives up and returns 0.0 with a report saying so. Don't score a 4-frame test.

A flat peak is not a finding. The report prints a confidence value; under about 0.10 the lag peak is too broad to trust, and the node says as much. Treat the offset as a hint and check a shot with hard plosives by eye before you commit to it.

It correlates articulation energy, not phonemes. It will reliably catch "this audio did not drive this video" and "the video is N frames late". It cannot certify that individual vowels are right, so a scene that scores 0.30 can still have one word that looks wrong. And on a music bed, a good score only means the mouth moves in time with the beat.

The residual spread is real. Across a two-window long render the offset settled at −4 in window 1 and +1 in window 2 - about ±2 frames of slop. Calibrate per shot, not per project.

That's the trade for a node with zero dependencies that runs in a couple of seconds. For a hard verdict on whether a dub is worth shipping, it's enough; for frame-exact dubbing you still want a real aligner downstream.