Audio To Phonemes

Lip sync is the part of the audio stack where ComfyUI gets real fast. You've generated a voice, you have a face, and now the mouth has to move at the right moments - and "at the right moments" is a per-phoneme timing problem, not a per-word one. Audio To Phonemes hands you that timing: it runs the audio through Vosk speech recognition and extracts phoneme-level timestamps, which is the data any proper mouth-rig needs to hit a "b" or a "p" when the sound actually happens.

The mechanism is straightforward and local. Vosk is an offline speech recognizer (no API, no key - everything runs on your machine), and this node feeds it your audio, gets back word and phoneme alignments, and packs them into a PHONEME_DATA output plus a duration float. It's the missing measurement step for lip-sync pipelines: AudioLength tells you the clip length, this tells you where inside it the speech sounds occur.

What you set

• audio - required, the AUDIO input (waveform + sample rate). That's the same AUDIO type your audio loaders and TTS nodes output, so it wires straight in.
• model_size - small (default, ~50MB) or large (~1.8GB). Small is accurate enough for most driving; large buys better accuracy on messy audio. This is the only real tradeoff you control.
• language - currently en is the only choice. Keep that in mind if your content is multilingual; the pack hasn't expanded this yet.

The models: what downloads

This is the part people miss. The node doesn't ship Vosk models - it downloads them on first use to ComfyUI/models/vosk/, from alphacephei.com. The tooltip is honest: small is 50MB, large is 1.8GB. First run after selecting large will stall while it pulls the model, so pick your size before you queue a batch of clips, not during.

# If the auto-download ever fails, grab the model manually:
cd ComfyUI/models && mkdir -p vosk
# unzip vosk-model-small-en-us-0.15.zip into models/vosk/

Where it hurts

Two realistic gotchas. First, the vosk pip package is a hard dependency of the pack - the requirements.txt installs it - but if you installed TrentNodes before the pack added it, pip install -r requirements.txt again or pip install vosk will sort it. Second, audio that arrives at a weird sample rate gets resampled by the node's audio preprocessing, so very low-quality captures can come back with muddy phoneme boundaries. If your mouth flaps look late or early, it's almost always the phoneme timing being off - check that your driving stage is consuming phoneme_data (the custom type) rather than trying to guess from word timestamps.

There's a broader honest caveat from the ecosystem: local lip-sync tooling is patchier than the image stack (see the audio layer notes in the KB), so treat this node as the measurement half - you'll still need a face-driving stage that consumes PHONEME_DATA. Where that stage exists in your graph, this node is the correct way to feed it.

Install

It's part of TrentNodes: ComfyUI Manager (search "Trent Nodes"), or

cd ComfyUI/custom_nodes
git clone https://github.com/TrentHunter82/TrentNodes.git
cd TrentNodes && pip install -r requirements.txt

Then restart ComfyUI. First phoneme run pays the model download; after that it's pure CPU inference and it's fast.