TBDub Generate (video + new speech)

What it is

This is the node the pack exists for. Bring a video of one person talking and a different speech track; you get back the same person, pose and background, with the mouth re-animated to the new words and that speech as the audio.

Be clear about what it is not: no translation, no TTS or voice cloning (bring your own speech), no animating a still into a talking head. Visual dubbing means the video already exists and you're changing the performance.

That distinction is the reason to care. The loud part of the local lip-sync scene is audio-driven generation - InfiniteTalk, LTX-2's native audio, Wan S2V - which invents a talking head from a still. TBDub does the narrower job people have files for: you shot it in the wrong language, or want a different read of the line. The one thread that even uses the phrase "visual dubbing" got LatentSync and a pile of research names back - plus the admission that it isn't turnkey.

How it works

Underneath is a Wan 2.2 stack: the upstream Wan 2.2 VAE plus a 2-step distilled Student DiT - ~6.3 B parameters, a 12.6 GB checkpoint - conditioned on HuBERT-large audio features (FP32, layers 9–12). The model never sees the whole frame. In full_frame mode MediaPipe finds the face on the CPU, the 512×512 crop is dubbed, and the crop is pasted back. The person, pose and background are literally the original pixels.

The recipe follows the official V1.1 Student setup: 2 steps, sigma shift 1.0, motion taken from the source latents, a pre-roll on the first clip. Long speech is split into clips on the official grid and clip i uses seed + i, so a fixed seed gives a reproducible multi-clip dub. That distillation is why it's quick: 98 seconds for a 5-second 1280×720 clip on a 5090 - the author's number for one machine and one input, not a benchmark.

Inputs and outputs

Required: runtime (wire a TBDub Runtime node), video (one person, face visible; its own audio is discarded), audio (the new speech), face_mode and seed (42 by default, matching upstream).

The two that need thought:

• face_mode - full_frame is the useful one: MediaPipe detects, the crop is dubbed and pasted back. cropped skips detection and resizes to 512×512.
• multiple_faces - refuse by default, so the pack stops if any frame shows more than one face. highest_score restores upstream's most-confident-face behaviour.

Optional: start_frame (skip source frames - the speech still starts at 0), allow_pingpong (repeat the source forward/backward if it's shorter than the speech; off means refuse) and fps_policy (resample_to_25 converts other rates with ffmpeg and reports it; refuse stops).

Two outputs. video is the dub, with the new speech muxed in whole as AAC - wire it into Save Video or Video Combine. report is a JSON string with the plan, per-clip timings, face stats, patches and memory numbers; when a job behaves weirdly, look there.

One README rule: output length is floor(speech_seconds × 25) frames, so the speech can run one frame (40 ms) past the last video.

Install

cd ComfyUI/custom_nodes
git clone https://github.com/hiroki-abe-58/ComfyUI-TBDub

Or ComfyUI Manager, search TBDub (publisher hiroki-abe-58). The node installs nothing into ComfyUI's environment - dependencies = [] on purpose - but it does require ComfyUI ≥ 0.38.0, for the current comfy_api video types.

The heavy half is the runtime, built once outside ComfyUI: a Python 3.10 venv with torch==2.9.0+cu128, mediapipe==0.10.21, transformers==5.1.0, the upstream checkout at commit 19e18e4, and the weights - tbdub_student.safetensors (12.6 GB), Wan2.2_VAE.safetensors, hubert-large-ll60k/, face_landmarker.task, config.json, null_prompt_emb.pt. Register it in ComfyUI/user/tbdub.runtimes.json, restart, load workflows/api/tbdub_dub_video.json. The Teacher checkpoint isn't needed.

Where people get burned

The runtime config is the whole ballgame. Empty dropdown, or "TBDub runtime 'x' is not configured"? Nothing else you do matters until that's fixed. Run TBDub Doctor first - cheapest minute you'll spend.

"memory admission refused." The pack won't start a job whose predicted memory commit doesn't fit, and kills one that sits at 95% for five seconds. On a busy 64 GB machine that fires for real. Switch to decode_mode=frame_chunks and memory_mode=stream: roughly half the decode memory, for an output that isn't bit-identical to upstream.

More than one face = refused. By design, and the most likely reason a good clip bounces. Crop to the one person you care about, or set multiple_faces to highest_score.

Non-25 fps footage gets resampled behind your back, reported in the report if not in your face. Set fps_policy to refuse if you'd rather be told.

Everything else is a hard limit with a clear message: 20 s of speech, 1920×1080, 750 frames, 1 GiB source file - all per-runtime settings in the config file, and raising them is your call. Cancel works between clips, not mid-clip, and a timeout tears down the whole runtime process tree.