LipSync GAP Model Loader

The LipSync GAP Model Loader is the least glamorous node in Geekatplay Studio's ComfyUI-LipSync-GAP pack, and also the one that decides whether your run works at all. It's the node that loads everything the lip-sync sampler needs: ByteDance's LatentSync UNet, a Whisper audio encoder, and your VAE, bundled into a single PERFECT_SYNC_MODELS object. If you're new to this pack, this is where you'll make the two choices that actually matter - Whisper size and precision - before the fun part starts downstream.

What it is and why you'd reach for it

This pack is a ComfyUI port of ByteDance's LatentSync, one of the "components open, products closed" releases ByteDance keeps feeding the local ecosystem (Apache 2.0, no API key, no cloud round-trip - the name is a lie in the best way). LatentSync is the audio-driven lip-sync pipeline that this pack's README calls v1.6 core. The loader is the entry point: no loader, no models, and the sampler just sits there unplugged.

It's a two-node pack, really. This loader plus the LipSync GAP Sampler, and everything else in the repo is legacy aliases of those two. Learn these and you've learned the whole thing.

How it works

Under the hood the loader does three jobs in one load_models call:

1. Loads the LatentSync UNet from ComfyUI/models/latentsync/ (expects latentsync_unet.pt), cast to the precision you pick.
2. Loads a Whisper checkpoint from models/latentsync/whisper/ and wraps it as the Audio2Feature encoder that turns speech into the conditioning the diffusion model lip-syncs to.
3. Wraps your ComfyUI VAE so the LatentSync pipeline can use it.

The precision dropdown is a straight dtype mapping - fp16 and fp8_e4m3fn for a real VRAM win on 12 GB cards, fp32 if you've got headroom and want maximum fidelity. Worth knowing: the Whisper dropdown does "alias handling," so picking tiny or tiny.pt lands on the same file.

The inputs that matter

• whisper_model - the one dial that changes results. Default tiny, up to medium/large. README's own guidance: bigger Whisper = stronger mouth articulation. Start tiny to test, bump up when the sync feels mushy.
• precision - fp16 default; switch to fp8_e4m3fn the moment you see OOM.
• unet_model - dropdown of whatever's in your models/latentsync folder. One UNet ships; if the dropdown is empty, the model download didn't happen.
• vae - wire any VAE Loader in (an SD 1.5 VAE works; the sampler casts it to match).

Output is a single models socket (PERFECT_SYNC_MODELS) that feeds straight into the sampler's models input. That's the whole circuit.

Installing it

Same install for the whole pack. ComfyUI Manager: search LipSync GAP. Or the manual route, then restart:

cd ComfyUI/custom_nodes
git clone https://github.com/GeekatplayStudio/ComfyUI-LipSync-GAP
cd ComfyUI-LipSync-GAP
install.bat

The installer locates ComfyUI Portable's python_embeded (or your system Python), pip-installs the requirements (diffusers, mediapipe, DeepCache, torchaudio, soundfile, ffmpeg-python - a real dependency stack, not a toy), and tries to auto-pull latentsync_unet.pt, whisper/tiny.pt, and the Mediapipe face_landmarker.task. If the model pulls fail it prints manual paths.

Common issues

• Model not found: the UNet and Whisper checkpoints live in ComfyUI/models/latentsync/, not the node folder. Installer creates the folders for you; verify files are actually there.
• Only tiny auto-downloads: the installer fetches whisper/tiny.pt specifically. If you want medium or large for stronger sync, you'll be downloading those by hand.
• VRAM errors: flip precision to fp8_e4m3fn before buying new hardware.
• Everything shows as missing after install: restart ComfyUI - the category appears as Geekatplay Studio/LipSync GAP and the folder must be exactly custom_nodes/ComfyUI-LipSync-GAP.

The pack is flagged beta and under active development, so rough edges are expected. When a run goes sideways, the loader is where you'll check first - it's the only node that can genuinely stop the graph before it starts.