Muse Voice Swap V1

Video models like LTX don't make sound, and the audio you bolt on afterward usually has one weak link: the voice. The ambience is fine, but the dialogue is whatever generic synthetic voice the audio model hallucinated - and it's the first thing a viewer notices.

Muse Voice Swap V1 exists to fix exactly that. Feed it your mixed audio and a 5–30 second reference of the voice you actually want, and it re-clones every spoken line in that voice, then lays the new vocals back over the untouched ambience. Same clip, same room tone, different person talking. Fully local, no API key, no quota.

How it works

Under the hood it's an eight-step pipeline, and your settings map onto the steps:

1. Separate - Demucs splits vocals from the bass/drums/other bed.
2. Detect - a VAD finds speech segments in the vocal track.
3. Transcribe - Whisper writes each segment's text.
4. Tag (optional) - a local Gemma model listens to each segment's actual audio and inserts Fish S2-style emotion tags like [excited] or [whisper].
5. Assign (optional) - ECAPA-TDNN compares each segment against your reference voices to guess who's speaking.
6. Clone - Fish Audio S2 re-speaks each line in the target voice.
7. Time-match - the clone gets placed back at the right moment.
8. Remix - the new vocals go over the untouched background, plus isolated stems.

The notable thing is step 0: this node doesn't reimplement any of it - it bridges into Christian Byrne's audio-separation-nodes-comfyui and Saganaki22's ComfyUI-FishAudioS2 at runtime and calls their classes. That means those two packs must be installed first - otherwise you get a "is ... installed and loaded?" error instead of a swap.

The timeline widget is the real differentiator

Most voice-swap tools are fire-and-forget. This one embeds a Muse-Director-style timeline widget right in the node: drag boundaries, fix transcript text, assign a speaker, tweak or clear the emotion tag, lock a segment, even drop a per-segment voice override (a filename in ComfyUI/input). Then flip use_timeline_override on and re-run - it skips re-detection and re-transcription and only re-clones the parts you changed. That edit-then-regen loop is what makes it usable on real clips instead of demos.

Inputs that matter

• source_audio and reference_audio are the only two you must wire. The reference should be clean, just the person talking.
• multi_speaker_mode (with reference_audio_2/_3) turns it into a 3-speaker dubbing rig and adds the speaker_2_vocals_only/speaker_3_vocals_only outputs. Add auto_assign_speakers and it guesses who says what - but treat the guess as a draft; short segments score low confidence regardless.
• use_llm_emotion_tags is the toggle behind the emotion-tag step - the fix for flat, monotone clones - and needs a Gemma checkpoint in models/text_encoders/ (the README points at gemma4_e4b_it_fp8_scaled.safetensors).
• fish_model_path - set it to s2-pro-fp8 (auto download) and the model fetches itself on first run.
• time_match_mode (ripple/stretch_to_fit/off) and bg_extend_mode (hold_last/loop/trim) decide how the clone lands back in time and what happens when the background runs short. Defaults are fine to start.
• vad_method defaults to Silero, the good one; energy_threshold is the older librosa path, kept as a fallback. The README's "energy-based VAD" line is stale - the shipped default is Silero.

Outputs

final_audio is your mix. new_vocals_only and bg_audio_only are the stems - handy if you want to rebalance or re-process them. And timeline_data should be wired back into the node's own timeline_data input so your manual edits persist across runs.

Install

Via ComfyUI Manager, search "Muse Collective Voice Swap" (or install by Git URL), or manually:

cd ComfyUI/custom_nodes
git clone https://github.com/muse-collective-26/MuseVoiceSwap.git

Then restart. The pack's own requirements (librosa, openai-whisper, soundfile, speechbrain, hyperpyyaml) install through Manager. Before you can run anything you also need, from Manager:

• ComfyUI-FishAudioS2 - the voice cloner
• audio-separation-nodes-comfyui - Demucs separation and the mixing
• ComfyUI-VideoHelperSuite - to load the audio/video in the first place
• A Gemma checkpoint in models/text_encoders/ if you want emotion tags

First run is slow and that's not a bug: Demucs (hdemucs_high_trained.pt), the Whisper model, the ECAPA-TDNN speaker encoder, and the Fish S2 pro model all auto-download on first use.

Gotchas

• Missing pack = immediate error. The bridge resolves classes at runtime, so install order matters. "AudioSeparation not found" or "FishS2VoiceCloneTTS not found" is the dependency, not the node.
• Duration drift. Fish S2 has no native duration control, so in ripple mode (the default) the final length can drift from the source over many segments. It's logged to the console, not hidden. stretch_to_fit trades that for possible time-stretch artifacts.
• Expect to hand-edit segments. No VAD nails boundaries on every clip - that's what the timeline is for. Plan a first pass that's just detection, correct, then re-run with the override on.
• Demucs on foley. It's trained on music stems; its separation of non-musical ambience is "unproven beyond ear-testing" per the author. On weird sources, check bg_audio_only before you trust the mix.

It's a lot of moving parts for one node and the prerequisite stack is real. But the payoff - a flat generated voice turned into a re-voiced mix with the room intact - is something nothing else in the ComfyUI audio corner does in a single box.