LipSync GAP Sampler

This is the node that does the actual work. Where the LipSync GAP Model Loader quietly assembles ByteDance's LatentSync weights, the LipSync GAP Sampler is where your video frames finally learn to talk - you feed it the PERFECT_SYNC_MODELS bundle, a batch of video frames, and an audio track, and it runs a latent diffusion pass that resculpts the mouth to match the speech.

What it is

Geekatplay's port of LatentSync's inference loop, wrapped for ComfyUI. It sits in the audio-driven-video corner of the ecosystem - the KB's framing is right: this isn't a model that generates audio, it consumes your audio as the driving signal and makes a face move to it. In a typical workflow it sits after your TTS (or a voice track from anywhere) and after whatever produced your video frames, and it hands synced frames + the audio out the other side for encoding.

How it works

The mechanism, from the source: it converts your IMAGE tensor to uint8 video frames, resamples the audio to 16 kHz mono (Whisper's native format), writes a temp WAV, and uses Mediapipe's face landmarker to find the mouth region. A feathered mask focuses the diffusion on that area - this is the "smart masking" the README brags about, and it's what keeps the rest of the frame from wobbling. The Whisper features condition a UNet3D latent diffusion over the masked frames with a DDIM scheduler, and out come resynchronized frames. The audio output is your input track passed back through, so you can mux it with the new frames.

The inputs that actually matter

Most defaults are sane; these four are the ones you'll touch:

• steps - default 20, README suggests 20–30. Fewer is faster but sloppier sync.
• guidance_scale - default 1.5. This is your "how hard should the mouth track the audio" dial; crank it up if the articulation feels lazy, but too high and it gets twitchy.
• frame_rate - match this to your source video. The README calls it out explicitly, and it's the classic cause of "the words don't line up with the lips."
• video_resize_limit - default 1280. This is your VRAM escape hatch: the node auto-downscales high-res footage to the limit (512/768/1024/1280 or None). It even rounds dimensions to even numbers so video codecs don't choke. 12 GB VRAM and fp8 still OOM? Drop this to 768.

The seed gives you reproducible runs; audio_offset_seconds and audio_duration_seconds let you sync a subsection of a track without pre-editing the file - set offset to 2.0 and it starts syncing 2 seconds in.

Outputs are images (the synced frame batch - wire it into a VHS-style video encoder) and audio (feed it to your audio-save node, or into a mux).

Installing

Same pack install as the loader: ComfyUI Manager search LipSync GAP, or git clone https://github.com/GeekatplayStudio/ComfyUI-LipSync-GAP into custom_nodes/, run install.bat, restart. The installer pulls the UNet, Whisper, and the Mediapipe face landmarker; the sampler needs all three present or it'll die at the mask step.

Where people get burned

• Corrupt or gray frames. This is the historical LatentSync-in-ComfyUI reputation - community threads are full of "why does LatentSync do this" with broken output. The source here even disables its optional DeepCache acceleration to debug a "Gray Square" issue. If you hit artifacts, drop video_resize_limit to something conservative and keep Whisper on tiny while you debug. DeepCache is off by default, so you're not leaving speed on the table by default.
• Audio too flat. If lips move but articulation is weak, it's almost always the Whisper size - switch the loader to medium/large, not this node.
• OOM. fp8 precision on the loader, then video_resize_limit, then shorter clips. In that order.
• Nothing comes out. Confirm the Mediapipe face_landmarker.task actually downloaded into the node's latentsync/weights/ - the mask step silently depends on it.

It's a beta pack, so treat the first run as a smoke test: load examples/workflow_lipsync_gap.json, short clip, tiny Whisper, and only then dial up. When it behaves, it's a genuinely good talking-head pipeline on hardware you already own.