Creature Lip Sync

Every lip-sync tool you'll find is built for faces - a person, a talking head, maybe a character with an actual mouth rig. So what do you do when the thing that needs to talk is a dragon, a mascot, or a cartoon animal whose mouth is just a painted oval? You paste cartoon mouth shapes over it, frame by frame. That's the whole job, and Creature Lip Sync (the "Creature Lip Sync" node) is the all-in-one version of it.

This node is the condensed form of TrentNodes' full lip-sync pipeline (Audio To Phonemes → Phoneme To Mouth Shapes → a compositor). Instead of chaining four nodes, you feed it audio, the video frames, and your nine mouth images, and it does the speech recognition, the mapping, and the compositing in one pass. It's the node you reach for when you want a talking creature with minimum wiring - not when you need pixel-perfect tracking (that's the tracked compositor's job).

How it works

The mechanism is the classic animation pipeline, compressed into one node:

1. Your AUDIO is run through Vosk speech recognition. The Vosk model auto-downloads on first use - small is enough for most clips, large is slower and more accurate on noisy audio.
2. Phonemes are mapped to the nine standard animation mouth positions: A–H plus X for silence, indices 0–8 in your mouth_shapes batch.
3. Each frame's shape is pasted onto the video at position_x/position_y, scaled by scale, and blended with blend_mode (alpha, replace, multiply, or screen).

Two knobs matter more than the rest. hold_frames (default 2) forces each mouth shape to stay up for a minimum number of frames - without it, fast speech makes the mouth vibrate between shapes. fps (default 24) must match your actual footage or the mouth timing drifts off the audio.

The inputs you'll actually set are frames (your video), audio (speech you want analyzed), mouth_shapes (9 images, A–H + X), and the position. Everything else - mask for blend-region control, feather for soft edges, scale, model_size - is optional and safe to leave alone until you need it. Outputs are the composited frames, plus mouth_sequence and phoneme_data if you want to inspect or reuse the intermediate data downstream.

Install

Search "Trent Nodes" in ComfyUI Manager and install, or:

cd ComfyUI/custom_nodes
git clone https://github.com/TrentHunter82/TrentNodes.git
cd TrentNodes
pip install -r requirements.txt

The pack's requirements include vosk, so this node's only real dependency is the speech model that downloads on first run. Restart ComfyUI after installing. One heads-up: the author has acknowledged ComfyUI Manager install hiccups tracing to an early repo rename that left a duplicate in the registry - if Manager flags the pack, the manual clone above is the reliable route.

Common issues

• Missing mouth shapes. The node expects a batch of 9 images in A–H + X order. Missing or mis-ordered shapes produce either an error or wrong phoneme-to-image pairing. Keep the files named A.png … H.png, X.png so you can verify the order.
• Timing drift. If the mouth is late or early against the audio, fps is wrong. Set it to the actual frame rate of frames, not the node's default.
• First run is slow. The Vosk model download can stall a workflow the first time; that's a one-time cost, not a bug.
• Rapid flapping. Raise hold_frames before reaching for anything fancier.

For static characters this node is a one-stop shop. For creatures that move around the frame, its fixed position_x/position_y won't cut it - that's what the pack's tracked compositor (Mouth Shape Compositor Tracked) is for, which takes a tracked point per frame instead of a single position.