LatentSync Node

LatentSync is ByteDance's open lip-sync model: you hand it a video of a face and an audio track, and it rewrites the mouth so it matches the speech. This node is an unofficial ComfyUI port by a solo dev (hay86) that wraps the whole official pipeline in a single D_LatentSyncNode - video and audio in, a resynced mp4 out. No API, no key, just local compute.

Be honest about what this is before you install it. It's a thin shell: the node doesn't run inference itself, it resamples your audio, then shells out to the pack's own scripts/inference.py with PYTHONPATH pointed at the pack folder. Every run re-imports torch and reloads the UNet (checkpoints are cached), so each job is slow to start and model state never persists. It works - it's just not the slickest integration in the ecosystem, which is roughly what the subreddit will tell you if you search "latentsync comfyui".

What it does under the hood

The pipeline is the official LatentSync one: audio gets resampled to 16 kHz, encoded by a Whisper-tiny audio encoder, and fed as cross-attention conditioning into a UNet3D latent-diffusion model. A DDIM scheduler runs 20 steps at guidance scale 1.0, fp16, and the result gets decoded through sd-vae-ft-mse. Worth knowing before you get excited: as bundled, this port runs the second_stage.yaml config - 256×256 and 16 frames - so it's soft, and only about a second of footage per pass. The official LatentSync runs higher-res configs; this one doesn't, and that's part of why people drift toward Wan-based S2V for production work.

It's also CUDA-only, full stop. The pipeline hard-codes device="cuda", so no CPU, no Apple Silicon.

The inputs that matter

There are only three, which is the whole appeal:

• video_path (STRING) - an actual file path on disk to the source video, e.g. C:\Videos\me.mp4 or /home/you/Videos/me.mp4. This is a hard path, not a video object: the node literally does assert os.path.exists(video_path), so you can't wire a generator's output straight in - you need a saved file first.
• audio (AUDIO) - a ComfyUI audio object, the kind you get from a loader like VHS_LoadAudio. The node resamples it to 16 kHz for you, so don't pre-process.
• seed (INT, default 1247) - the noise seed for the diffusion pass. Same video + same audio + same seed = same result.

The single output, also called video_path, is a STRING pointing at the finished file in ComfyUI's output/ folder (something like latentsync_abcde_out.mp4), with a matching _mask.mp4 alongside it. Since it returns a path rather than a video stream, preview it with a text/viewer node or just grab the file from the output folder.

Installing it

ComfyUI Manager is the easy route - search latentsync and let it pull the repo plus requirements. Manually:

cd ComfyUI/custom_nodes
git clone https://github.com/hay86/ComfyUI_LatentSync
cd ComfyUI_LatentSync
pip install -r requirements.txt

Two setup gotchas up front. First, the README explicitly wants ffmpeg and libgl1 installed (sudo apt install ffmpeg libgl1 on Debian). Second, requirements.txt is the full training stack - torch with CUDA 12.1 wheels, xformers, diffusers, decord, mediapipe, librosa, face-alignment - heavy, and it can clobber an existing torch install without a venv. The pack also ships a setup_env.sh if you'd rather script the environment.

First run and troubleshooting

First run downloads latentsync_unet.pt and whisper/tiny.pt from the chunyu-li/LatentSync HuggingFace repo into the pack's own checkpoints/ folder, plus the VAE from stabilityai/sd-vae-ft-mse. That's a multi-gigabyte download, so don't panic when the first queue takes ages - it's fetching, not frozen. If the auto-download fails, grab the files manually from that HF repo and drop them in the right layout.

Where people actually get burned:

• Python version. LatentSync's stack expects Python 3.8–3.11, and ComfyUI portable ships newer runtimes. The common failure is someone on Python 3.12 whose LatentSync install throws obscure import errors - the fix is running ComfyUI on 3.11 (a second portable install works fine).
• Errors hide in the console. Inference runs via os.system, so pipeline exceptions never surface as a nice red ComfyUI error - they show up in the terminal (the node prints the exact python command it runs). A run that dies silently? That's the first place to look.
• Slow, then done. Start-to-finish is minutes per clip on a midrange card. Budget for it.

Is it worth it? If you specifically want LatentSync's output style inside ComfyUI and don't want to babysit the official repo's scripts, yes - this is the least-effort way to get it, and 3 impressions means almost nobody has found it yet. If you just want lip-sync and don't care whose model does it, the community's current favorites are Wan S2V and InfiniteTalk, which integrate better and don't cap you at 256×256. For LatentSync-the-model's sake, this node does the job - with the caveat that "the job" is specific, slightly dated, quite charming.