🗣️ Silent Speech Analyzer

This is the odd one out in the pack, and kind of a clever idea. 🗣️ Silent Speech Analyzer watches a video of someone talking - with no audio, or where you're ignoring the audio - detects when the mouth is moving, and produces subtitle timing from that. The point isn't transcription; it's timing. You get an SRT scaffold with accurate start/end times for each speech burst, which you then fill in with real text and hand to the suite's TTS SRT nodes to generate perfectly-timed dubbed audio. For dubbing, animation reference, and lip-sync work, it solves the tedious first step: figuring out when speech happens.

Be clear-eyed about it, though - the README calls the viseme/word-guessing "experimental approximations," and it means it. Treat this as a timing tool, not a lip reader.

How it works

It runs face and mouth-landmark detection frame by frame (via Google's MediaPipe, the stable path), measures mouth openness/movement over time, and groups the moving stretches into speech segments with start and end times. On top of that timing it can guess placeholder text - approximate visemes (vowel/consonant shapes) mapped through the CMU pronouncing dictionary - but those are phonetic filler, not real words. The three-level pipeline (frame movement → syllable grouping → word prediction) is really there to give you a scaffold you then edit by hand.

The inputs and outputs that matter

• video - the clip to analyze (VIDEO input).
• provider (MediaPipe, OpenSeeFace, dlib, default MediaPipe) - the face-detection backend. MediaPipe is the one to use; the README flags OpenSeeFace as experimental and not for production (it's the fallback for Python 3.13, where MediaPipe won't install). Only switch if MediaPipe isn't available.
• sensitivity (0.05–1, default 1) - how readily it counts mouth motion as speech. Lower it if it's catching every twitch; raise it if it's missing quiet talking.
• min_duration (default 0.05s) - the shortest segment it'll emit, filtering out micro-blips.
• output_format (SRT, JSON, CSV, AUDIO_REGIONS, default SRT) - SRT for subtitle workflows; the others for data/analysis.
• srt_placeholder_format (Words, Syllables, Characters, Underscores, Duration + Length) - what filler to drop into each cue, since you'll be replacing it anyway.

Outputs include srt_output (the timing scaffold), timing_data (a structured TIMING_DATA for downstream nodes), a preview video with detection overlaid, plus movement_frames and confidence_scores lists. merge_threshold and confidence_threshold (optional) tune how aggressively nearby segments merge and how confident a detection must be to count.

Installing it

Ships with TTS Audio Suite. Easiest: ComfyUI Manager → search TTS Audio Suite → install → restart, which runs the pack's install.py - and this node is the reason that script matters: it handles the MediaPipe → OpenSeeFace fallback on Python 3.13 automatically. Manual: cd ComfyUI/custom_nodes && git clone https://github.com/diodiogod/TTS-Audio-Suite.git, then python install.py from inside the folder with your ComfyUI venv active (run the script, not just a bare requirements install). Linux: portaudio19-dev libsamplerate0-dev first. MediaPipe pulls its own model assets on first run.

Common issues

• MediaPipe won't install (Python 3.13). Known incompatibility. The installer falls back to OpenSeeFace automatically, but OpenSeeFace is experimental - if you can, run this on Python 3.12 with MediaPipe for reliable results.
• The predicted words are gibberish. Working as intended. The text is a phonetic placeholder, not a transcript - you're meant to replace it. Use this for timing, then write the real lines yourself.
• Missing or spurious segments. Tune sensitivity (down if it's over-triggering on non-speech motion, up if it's missing soft talking) and min_duration to drop blips. confidence_threshold helps in busy footage.
• Face not detected. MediaPipe needs a reasonably clear, front-ish face. Extreme angles, occlusion, or tiny faces in frame tank detection. Crop closer to the speaker if you can.
• Expecting finished subtitles. It gives you a base timing template. The content editing is manual, always.