X-Dub Lip Sync (Frames Compatibility)

X-Dub has two doors into the same engine. X-Dub Lip Sync (Frames Compatibility) is the one for pipelines that live in IMAGE batches - which, if you've used Video Helper Suite, is every video pipeline you've ever built. It takes frames plus an audio track, re-lips a single person so their mouth matches the speech, and hands frames back to you. Same model, same result as the Video node, different wiring.

Here's the honest version of when to reach for it: your upstream graph is already frame-based. Video Helper Suite loads a video to an image batch, you might be doing frame-level editing or a pose pass in between, and you don't want to round-trip out to a file and back. This node takes your IMAGE batch, turns it into a temp MP4, runs the whole X-Dub pipeline, and returns frames (plus the original audio and the rendered MP4 path). The README's example workflow is exactly this: VHS load → frames node → Video Combine.

What actually happens

The heavy lifting is done by an isolated runtime the pack installs - the same one the Video node uses. DWPose (ONNX, runs on CPU/GPU) tracks the face and body, Whisper transcribes the audio, Wav2Vec2 pulls out speech features, and the X-Dub fine-tune of Wan 2.2 TI2V-5B re-denoises the footage so the lips match. Long audio gets split into overlapping 77-frame clips with a 5-frame overlap, and each clip runs the full denoising loop. Audio in, mouth moves. Reference in, the person stays the person.

Inputs that matter

Four things to get right, then three dials you'll touch maybe twice:

• images - your IMAGE batch. Must be a non-empty [frames, height, width, 3] tensor; the node errors otherwise.
• audio - the AUDIO track that drives the lips. Usually a TTS clip from Chatterbox or similar, or the original soundtrack you want dubbed.
• vae - a standard ComfyUI Load VAE node, but it has to be the Wan 2.2 VAE (48 latent channels). The node checks and refuses to run otherwise.
• xdub_model - from the Load X-Dub Model node.

The dials: ref_cfg_scale (default 2.5) is how hard the output holds onto the original video's identity and structure; audio_cfg_scale (default 10) is how strongly the mouth follows the speech. If lips look lazy, nudge the audio scale up a couple points. num_inference_steps (default 30) trades quality for runtime. seed controls the noise; set it for reproducible takes.

Outputs

• images - the re-lip-synced frames, same shape as your input batch.
• audio - your input audio passed straight through.
• video_path - a string path to the rendered MP4 in ComfyUI/output/x-dub/, in case you want the file directly.

The one gotcha that bites everyone

An IMAGE batch carries no frame-rate metadata. X-Dub always renders at 25 FPS. So when you recombine your frames downstream, you must set Video Combine to 25 FPS - otherwise VHS-style nodes fall back to 8 FPS and the whole clip plays in slow motion. The README spells out the trap: 1,013 frames is about 40.5 seconds at 25 FPS but 2 minutes 7 seconds at 8 FPS. That's not X-Dub generating extra speech - it's playback speed. It's the single most common reason people think the model is broken.

Installing and the heavy stuff

Install is shared with the whole pack:

cd ComfyUI/custom_nodes
git clone https://github.com/DarkNoah/comfyui-x-dub.git
cd comfyui-x-dub
bash install_runtime.sh

Requires Linux + NVIDIA CUDA, uv, ffmpeg/ffprobe on PATH, and about 21 GB of VRAM. The script builds an isolated Python 3.10 venv so it doesn't disturb ComfyUI. Models come from the KlingTeam/X-Dub Hugging Face repo: the X-Dub DiT, Wan 2.2 VAE, UMT5-XXL text encoder, Whisper large-v2, Wav2Vec2, and the two DWPose ONNX files - arranged under ComfyUI/models/ as the README shows.

Common issues

• Wrong playback speed - the fps gotcha above. Set everything downstream to 25 FPS.
• The [VRAM] messages - offload/onload chatter in the terminal is informational. Only exceptions or CUDA out-of-memory errors are real problems.
• It's slow - a 40-second clip is roughly 14 diffusion clips at 30 steps each. That's the model, not a broken install.
• Flicker or identity drift - the public Wan-based model has known limits; single person only, and fast head motion makes face tracking unstable.

If you're starting fresh, the Video node is the cleaner path. But if your graph already breathes image batches, this is the door that doesn't make you rebuild anything.