X-Dub Lip Sync (Video)

This is the node you actually want. X-Dub Lip Sync (Video) takes a real video and an audio track, and re-renders a single person's face so their mouth matches the speech - dubbing, in the literal sense: new voice, same face, lips that keep up. It's the preferred path in the X-Dub pack because it uses ComfyUI's standard VIDEO and AUDIO types, so it slots into a modern ComfyUI graph without any frame-batch contortions. The frames-compat sibling exists for legacy pipelines; this one is the clean way in.

X-Dub wraps KlingAIResearch's model of the same name - a fine-tune of Wan 2.2 TI2V-5B, Alibaba's last open Wan and the backbone of the local video ecosystem. The dubbing angle is what makes it unusual: most audio-driven avatar work is either fast but shallow (head-only animation) or a poor local reputation (Wan S2V). X-Dub goes after the full thing: the person keeps their body, their motion, their background, and only the mouth gets re-animated to fit new audio.

How it works

The node normalizes your video to 25 FPS MP4 with ffmpeg, then hands it to an isolated runtime. Inside, three things happen:

• DWPose (two ONNX models - a YOLOX detector plus a whole-body pose model) finds the face and body so the re-generation stays anchored to the original person.
• Whisper large-v2 transcribes the audio and Wav2Vec2 extracts speech features. Together they tell the model what is being said and how it sounds.
• The X-Dub DiT re-denoises the footage conditioned on all of it. Long audio is split into overlapping 77-frame clips with a 5-frame overlap (72-frame stride), each clip running the full denoising loop, then the pieces are stitched, color-corrected, muxed with the audio, and written out as an MP4.

That clip splitting is why it's slow - and why the terminal looks scary. A roughly 40-second input is about 14 clips; at the default 30 steps that's around 420 denoising iterations before VAE decode.

Inputs that matter

• video - your source clip. Single person, face reasonably still; fast head motion makes tracking unstable.
• audio - the AUDIO track that drives the lips. TTS output, a foreign-language voiceover, whatever you want the face to say.
• vae - a Load VAE node with the Wan 2.2 VAE (48 latent channels). The node validates this and stops with a clear error if you feed it a Wan 2.1 or SDXL VAE.
• xdub_model - from Load X-Dub Model.

The dials: ref_cfg_scale (default 2.5) pins the output to the original video's identity; audio_cfg_scale (default 10) drives how strongly the mouth follows the speech - if the lips look half-committed, raise it a couple points. num_inference_steps (default 30) is quality vs. runtime. seed (default 42) for reproducible takes.

Outputs

• video - a VIDEO you can feed straight into downstream nodes.
• video_path - a string path to the finished MP4 in ComfyUI/output/x-dub/.

Installing it

Same install as the whole pack - it's one repo, three nodes:

cd ComfyUI/custom_nodes
git clone https://github.com/DarkNoah/comfyui-x-dub.git
cd comfyui-x-dub
bash install_runtime.sh

Requirements up front: Linux with NVIDIA CUDA and roughly 21 GB VRAM, uv on PATH, ffmpeg/ffprobe on PATH, and Python 3.10 (which uv provisions). The script builds an isolated venv inside the pack so its pinned torch/transformers versions never touch ComfyUI's environment - a courtesy that keeps your other nodes working. ComfyUI Manager should find it by searching "X-Dub". Restart ComfyUI after installing.

Then download the bundle from KlingTeam/X-Dub on Hugging Face and lay it out under ComfyUI/models/ per the README: X-Dub_model.safetensors in diffusion_models/, Wan2.2_VAE.safetensors in vae/, the UMT5-XXL text encoder and tokenizer in text_encoders/, Whisper and Wav2Vec2 in audio_encoders/, and the two DWPose ONNX files in dwpose/. (The pack also accepts these under its own models/ folder.)

Common issues

• "This ComfyUI version does not provide the standard VIDEO API" - your ComfyUI is too old for the VIDEO type. Update ComfyUI, or fall back to the frames-compat node.
• The [VRAM] offload/onload messages - informational. The Whisper and Wav2Vec2 wrappers even say VRAM management is "disabled"; that's expected. Only an exception or CUDA OOM is a real failure.
• OOM - 21 GB is a floor, not a suggestion. Shorter input reduces clip count; the model itself won't shrink.
• Flicker or identity/color drift - known limits of the public Wan-based model, not a setup error. One person, steady head motion, and modest clip length are your best defenses.
• Playback speed wrong on the frames path - if you switched to the compat node, remember X-Dub outputs 25 FPS and image batches carry no fps metadata.

The one thing to respect: this is a dubbing tool, and generated speech on a real person's face is exactly the kind of media that should be disclosed as generated. Within that, it's a striking example of what a Wan 2.2 fine-tune can do on a single card - slow, heavy, occasionally wobbly, but genuinely doing the thing.