IP_LAP Node

IP_LAP is a ComfyUI port of the IP_LAP research project that takes a video of a face and an audio file and makes the mouth say the audio. Think dubbing, avatar videos, new dialogue on an existing clip. It's a niche pick in the lip-sync space - SadTalker and LatentSync get all the community love, this one gets almost none. Zero reddit presence, a bilibili tutorial instead of a docs page, and a node description that is literally "hello world!". None of that means it's bad. It just means you're mostly on your own, and this article is your map.

What it actually does

Most lip-sync tools paste a synthesized mouth over your video. IP_LAP takes a different route: it re-renders the face. Under the hood (read ip_lap/inference.py) two models work in sequence. First, a landmark transformer takes the audio's mel spectrogram plus the pose landmarks (eyes, brows, nose - the parts that don't move with speech) from your source frames, and predicts the lip and jaw landmarks that belong with the audio. Second, a renderer draws a fresh face from those landmarks, guided by reference frames sampled from your video and their sketches. That generated face gets pasted back into the frame and blended with a mask.

The payoff: the mouth isn't swapped in, it's consistent with your source face's texture and lighting, because the renderer learns from reference crops of that same face. The cost: it's slow and frame-by-frame, and demands more from your hardware than a simple paste job.

The inputs that matter

The node takes an audio (AUDIO) and a video (VIDEO) - that's the whole job. The rest are dials you can safely leave at defaults until something breaks:

• T (default 5) - how many frames the landmark transformer processes as a batch. More context, slower.
• Nl (default 15) - reference landmark sets it samples, sorted by lip openness so it has closed-mouth and open-mouth examples.
• ref_img_N (default 25) - reference face crops for the renderer. This is the quality knob.
• img_size (default 128) - resolution the face is rendered at before being pasted back. Crank this for quality and you get slow renders and VRAM complaints instead.
• mel_step_size (default 16) - the mel-spectrogram window per frame. Don't touch it.
• face_det_batch_size (default 4) - batching for face detection. It halves itself on out-of-memory, which is a nice touch.
• checkpoints_path - where the .pth weights live; defaults to the pack's own weights/ folder.

The single output is a VIDEO, ready to wire into a preview or the pack's CombineAudioVideo node.

The gotcha nobody warns you about

The output video has no audio track. The code writes frames with OpenCV's mp4v codec and the ffmpeg mux step is sitting there commented out. The pack ships a CombineAudioVideo node (same AIFSH_IP_LAP category) precisely for this: feed it the vocal audio, a BGM, and the lip-synced video and it muxes them together. Or add audio yourself. Remember it, or you'll spend a confused ten minutes wondering where the sound went.

Two more things the source tells you up front: your video gets converted to 25 fps (ffmpeg must be on your PATH), and if the audio outlasts the video the frames repeat to catch up. It will also hard-error on any frame with no detectable face - not the tool for turning shots.

Installing it

ComfyUI Manager will find it if you search "ComfyUI-IP_LAP". Or the manual route:

cd ComfyUI/custom_nodes
git clone https://github.com/AIFSH/ComfyUI-IP_LAP.git
cd ComfyUI-IP_LAP
pip install -r requirements.txt

Then the part that bites: the weights aren't bundled. Download the two .pth files (landmarkgenerator_checkpoint.pth, renderer_checkpoint.pth) from the README's jianguoyun or OneDrive links, drop them in ComfyUI-IP_LAP/weights, and restart ComfyUI. Also make sure ffmpeg is callable from your command line - the README suggests WingetUI on Windows.

When it breaks

The one error with actual community footprint is ModuleNotFoundError: No module named 'torchvision.transforms.functional_tensor' - it's in the pack's own note.txt and it's the top reddit hit for this node. Newer torchvision moved that function; the fix is patching rgb_to_grayscale to import from torchvision.transforms.functional. On Linux you may also hit libGL.so.1 missing - that's your system's OpenGL library, not this pack; install libgl1 and rerun. Missing weights gives you a load error pointing at checkpoints_path, which means you skipped the manual download. And be honest about expectations: this is one developer's port of a research project, so when something breaks, the answer is usually the source code, not the subreddit.

If you just want to lip-sync something fast and don't need the re-rendered realism, LatentSync or SadTalker are the smoother road. If you want the mouth to genuinely belong to the face in the frame and don't mind the fiddling - this is the one.