Phoneme To Mouth Shapes

The middle step of any classic lip-sync: your audio has been chopped into phonemes with timestamps, and now you need to know which mouth shape to draw on which frame. That's this node. It takes the timestamped phoneme data, your video's frame rate and audio duration, and turns them into a per-frame list of mouth-shape indices - the exact MOUTH_SEQUENCE the compositors in this pack consume.

How the mapping works

Human speech makes about 44 distinct phonemes in English, but animation mouths don't have 44 drawers - they have 9. The node compresses phonemes into visemes (the visual equivalents: mouth shapes that look the same on screen) using a mapping_type of arpabet (the default), ipa, or simplified. Arpabet is the practical choice: it's what the upstream Audio To Phonemes node's Vosk model emits, so keeping the default means the whole chain speaks one dialect.

You tell it how long the clip is and what frame rate the video runs at (duration in seconds, fps), and it lays the phoneme timing onto the frame grid. From there, two controls shape the output:

• hold_frames - the minimum number of frames a shape must be held. Default 2. Without a minimum, a fast talker produces one-frame blips that flicker horribly in the final video.
• smoothing (default on) - removes flicker by cleaning up the held sequence. There's a reason the defaults are what they are; turn either off and you'll see why within a second of rendered mouth.

What comes out

Two outputs: mouth_sequence (the per-frame indices, 0–8 mapping to A–H plus X for silence) and frame_count (how many frames the sequence covers). Wire mouth_sequence into a Mouth Shape Preview to read the timeline as text before you spend any GPU time, or straight into a compositor when you're confident.

Getting the duration and fps right

These two inputs are where beginners lose time. duration should match the actual audio length - if Audio To Phonemes measured a 10-second clip but you type 8, the tail of the speech gets squeezed into silence shapes. fps must match the video you're compositing onto, or the mouth drifts out of sync over the length of the clip. If your mouth is synced at the start but late by the end, it's an fps mismatch, almost always.

Installing it

Part of TrentNodes (one pack, install once):

cd ComfyUI/custom_nodes
git clone https://github.com/TrentHunter82/TrentNodes.git
cd TrentNodes
pip install -r requirements.txt

Or ComfyUI Manager, search "Trent Nodes". Heads-up: the author renamed the repo on day one, which briefly created two registry entries and made Manager flag the pack as "unsafe" for some people - the clone above is the reliable fallback. The node itself needs nothing beyond the pack's base install; vosk (the phoneme extractor upstream) is the heavier dependency, and it's in requirements.txt with a model that downloads on first use.

When to reach for it

This is the "non-human character animation" path the pack's README describes - creatures, puppets, anything where you have drawn mouths and want them driven by real audio. If that's your goal, this is the right node. If you'd rather not assemble the five-node chain by hand, the pack's all-in-one Creature Lip Sync bundles analysis, mapping, and compositing into a single node - this one is for when you want the steps separate and debuggable.