DiffusionGemma Timed Lyrics Analyzer

Here's the gap that kills generated lip sync: your lyrics tell the model what to sing, but nothing tells it when. A generated singer mouthing words on guessed beats is worse than no lip sync at all. DiffusionGemma Timed Lyrics Analyzer exists to close that gap - it measures the actual vocal stem, transcribes it, and aligns the transcribed phrases to your authored lyrics so downstream systems know exactly when each line happens.

And it's engineered to fail safely, which is the part to respect. Lyrics analysis is notoriously unreliable - Whisper stumbles, vocals are muddy, alignment is weak. This node's contract is that any uncertainty produces a safe Natural/audio-led report rather than blocking your video or inventing singing. Confirmed instrumental intervals are recorded as explicit non-vocal evidence. The pipeline would rather have no lip sync than wrong lip sync, and honestly that's the correct trade for a tool chain where local song generation's vocals are still its weakest link.

How it works

The node is lazy, keyed off the canonical performance_mode string from DiffusionGemmaMusicVideoPerformanceMode. In Natural and Dance modes it short-circuits - its vocal-separator and Whisper inputs are never even evaluated. Only in Lyrics + lip sync mode does the real work happen:

1. Verify the incoming final_audio waveform hash matches the locked master_audio_sha256 (stale audio = no analysis).
2. Validate the local Demucs vocal stem and align it to the full mix.
3. Transcribe only the detected vocal intervals, in chunks no longer than 15 seconds, with Whisper.
4. Align those phrases monotonically to the authored lyric lines and emit a timing report locked to song hash, lyric hash, and excerpt window.

Any failure - missing stem, missing pipeline, silence, weak alignment, a model error - drops into the fallback report. It never blocks a queue.

Inputs and outputs

Required: final_audio (AUDIO), performance_mode, lyrics, master_audio_sha256, song_duration_seconds (default 90), excerpt_start_seconds, excerpt_duration_seconds (default 25), language (default en, with auto and a dozen more), and minimum_alignment_confidence (default 0.55 - lower it if alignment keeps failing on heavily accented vocals). Optional but required for real work: vocal_stem (AUDIO) and whisper_pipeline.

Outputs: timed_lyrics_report_json, a human status, safe_ready (the boolean you check before trusting it), and aligned_preview.

The heavy dependencies

This is where the pack stops being free. Lyrics mode wants two local models: a Whisper snapshot (whisper-large-v3-turbo as a complete folder at ComfyUI/models/whisper/whisper-large-v3-turbo - the reference workflow deliberately sets download_missing=False so a network change can't silently alter a repeatable run) and torchaudio's Demucs HDEMUCS_HIGH_MUSDB_PLUS checkpoint for separation. The whisper_pipeline socket expects a loader's MTB-style pipeline (the reference flow uses comfy-mtb's local Whisper loader), and the vocal stem typically comes from a Hybrid Demucs separator such as the one in comfyui_fill-nodes.

Install the pack from ComfyUI Manager (search "DiffusionGemma Prompt Builder") or git clone into custom_nodes. First time in Lyrics mode expect a download wait and a first-run hum. If safe_ready stays false, check the status - it'll name the exact fallback reason rather than leaving you guessing.