Zura · Mouth Motion from Audio

This is the mouth engine. Point it at a stack of face crops and an audio track, and it gives you back the same faces with their mouths moving to that audio.

It exists as a separate node because in this pipeline lip sync is a guide, not a finish. The pack's Wan route uses the animated face as pre-generation conditioning so the video model carries the new mouth performance through the render, and the Refine lips after generation option in Studio is the same engine used the other way round - as a post-render finish. Same node, two positions in the graph. That split is the sane way to do it: you can inspect the mouth motion as images before you spend twenty minutes sampling.

Inputs that matter

• model - the ZURA_SPEECH_MODEL from the speech model loader. That's where LatentSync 1.6 vs MuseTalk 1.5 gets decided.
• face_images - the crops. Not whole frames; the engine works on faces.
• audio - any AUDIO. Generated speech from your TTS node, or a reference track, or the audio lifted out of a video.
• fps (default 24, 1–120) - should match your clip. The audio conditioning is sampled at the real frame rate; forcing a 25 fps convention onto 24 fps footage is exactly the kind of thing that makes mouths slide half a frame.
• seed (default 42) - there's stochasticity in the synthesis, so this is your reproducibility handle. Change it when a take has a weird mouth, not the timing settings.
• batch_size (default 4, 1–16) - MuseTalk's batch size. The tooltip says sharp lip sync uses fixed 16-frame windows, so this is a throughput dial, not a quality dial.
• Optional: pose_data and face_boxes. Pass them if you have them. They let the node keep the face anchored to the head it belongs to and hold quiet frames at rest instead of guessing.

Output: speech_faces, an IMAGE batch. Wire it into the compose node, or preview it - honestly, preview it first.

How it works, and why the silence handling matters

Under the hood there are two paths. The MuseTalk path encodes the audio with Whisper-tiny into the 50 Hz feature stream the model was trained on, slides a ten-step window per output frame, and runs a UNet over the crop. The LatentSync path uses the vendored ByteDance network with its own ten-step, five-layer window at your actual fps.

The detail I'd actually praise: there's an explicit speech-activity gate. Ten-millisecond energy bins, a threshold relative to the track's own peak, and a causal 120 ms release so nothing moves before the speech starts. Quiet sections return a resting face. That is precisely the failure people report on raw lip-sync installs - "lips moving when there's no audio" - and it's handled here rather than left to you.

Install

The node ships with the pack; the models do not.

cd ComfyUI/custom_nodes && git clone https://github.com/ZURAVFX/ComfyUI_zura_nodes
pip install -r ComfyUI_zura_nodes/requirements.txt

Then run Setup_Speech_Windows.cmd once (or replicate it) to populate ComfyUI/models/zura_speech/. LatentSync 1.6 wants a CUDA GPU; MuseTalk 1.5 does not. See the model-loader page for the exact folder layout - the file-matching is literal.

Face detection is not this node's job. In the pack's own graph, native PoseAndFaceDetection from ComfyUI-WanAnimatePreprocess supplies the crops, boxes and pose data, and that pack has to be installed separately.

Common issues

"Speech guidance needs valid RGB face crops." You fed it something that isn't an image batch of face crops - a whole frame with a person in the corner will technically pass and then look awful, because the engine expects the face to fill a good chunk of the frame.

Frames out of step with the video. Check fps first, then check where the audio came from. The node pads or trims audio to exactly frames / fps seconds worth of samples; if the track you handed it starts at a different point than the clip, everything downstream inherits that offset.

Everything is a still face. That's the silence gate telling you the track's peak energy is low. Normalise or gain your speech up before it gets here - a voice track recorded 30 dB down will look like silence against a 2% threshold.

Mouth motion looks soft. MuseTalk is the softer engine. If you installed both and want the sharp route, force model to LatentSync 1.6 in the loader instead of leaving it on Automatic and hoping.