DiffusionGemma Music-Video Performance Mode

Here's the thing nobody tells you about generated music videos: the model will happily make your performer appear to sing - mouthing syllables that were never authored, on beats that don't match the words - and the result is uncanny in the worst way. The DiffusionGemma pack treats "what is the performer doing" as a single decision, and DiffusionGemma Music Video Performance Mode is where you make it. It sits before the Director and declares whether the subject should sing, dance, or simply move naturally, and - this is the clever part - it's the one authoritative answer shared by target guidance, lyric-window selection, timed-lyrics analysis, and H3 lane planning. No more singer in one lane and dancer in the next.

How it works

You give it a base_audio_guidance (your own notes about the video's musical direction) and pick a performance_mode. It compiles a mode-specific directive and returns it as target_audio_guidance - base notes plus the mode's rules. It changes prompt text only: the selected song, the conditioning waveform, and the pristine final soundtrack are never touched.

The three modes are worth understanding rather than flipping blind:

• Natural / audio-led sync (default) - adds no lyric schedule. The connected audio governs whether visible articulation happens at all; the model isn't pre-scripted to mouth words.
• Dance / music sync - deliberately suppresses singing, speaking, mouthing, and lip-sync while keeping beat-, phrase-, and dynamics-aware body motion. This is the mode for a choreography-first video where a lip-synced face would wreck the illusion.
• Lyrics + lip sync - supplies a bounded authored-lyric window only as lexical/pronunciation candidates for verified downstream timing. Notice the modern posture: it no longer commands a subject to sing every selected line. If verified timing is absent, it falls back to Natural behavior rather than inventing mouth motion.

Inputs and outputs

Inputs are base_audio_guidance (multiline string) and the performance_mode dropdown. Outputs: the canonical performance_mode string, the compiled target_audio_guidance, and a status line. Because downstream nodes key their behavior off the canonical string - DiffusionGemmaTimedLyricsAnalyzer won't even evaluate its Whisper and vocal-stem inputs unless the mode is Lyrics + lip sync - wire the output, don't retype it by hand. A typo'd literal reads as a different contract entirely.

Wiring and gotchas

Feed target_audio_guidance into the target/performance guidance that reaches the Director, and fan the performance_mode string out to the timed-lyrics analyzer and the H3 lane planner. Install via ComfyUI Manager (search "DiffusionGemma Prompt Builder") or git clone https://github.com/exportAnything/ComfyUI-DiffusionGemmaPromptBuilder into custom_nodes. One honest caveat: even in Lyrics mode this node only schedules the ambition - the actual timing must come from a validated timed-lyrics report, and the audio-led reality is that local song generation still struggles with clean vocals. Which is precisely why the pack fails safe toward "move naturally" instead of forcing fake singing.