MuseVoiceSwap

MuseVoiceSwap

A ComfyUI custom node for swapping the speaking voice in a mixed audio clip (dialogue + ambient sound) while keeping the background intact — built for post-processing audio generated by LTX-style video/audio pipelines, but works on any source audio.

Given a source clip and one or more reference voices, Muse Voice Swap V1:

1. Separates vocals from background (Demucs).
2. Detects speech segments in the vocal track (energy-based VAD), falling back to the whole clip as one segment if nothing is detected.
3. Transcribes each segment (Whisper).
4. Optionally auto-tags emotion/prosody per segment using a local Gemma model (fed the actual segment audio, not just the text) — fully local, no API key, no quota.
5. Optionally auto-assigns speakers per segment (multi-speaker mode) using ECAPA-TDNN speaker-embedding similarity against your reference voices — a best-guess classifier, meant to be reviewed/corrected, not a guaranteed answer.
6. Clones each segment's line with the target voice (Fish Audio S2).
7. Time-matches the clone back into place (ripple / stretch-to-fit / off).
8. Reassembles the final mix, plus isolated vocal/background/per-speaker stems.

All of this is editable through a Muse-Director-style timeline widget built into the node: review/fix transcripts, per-segment emotion tags, per-segment speaker assignment, and per-segment voice overrides, then re-run with use_timeline_override on to skip re-detection/re-transcription and only redo the parts you changed.

Requirements

This node bridges into several other custom node packages at runtime rather than duplicating their code — install these first:

• ComfyUI-FishAudioS2 — voice cloning
• audio-separation-nodes-comfyui — Demucs separation, time-stretch, mixing
• ComfyUI core's CLIPLoader + TextGenerate nodes (for the local Gemma emotion-tag step) — a Gemma checkpoint in ComfyUI/models/text_encoders/ (e.g. gemma4_e4b_it_fp8_scaled.safetensors)

Python dependencies (see requirements.txt): librosa, openai-whisper, soundfile, speechbrain, hyperpyyaml.

Inputs / Outputs

Inputs: source_audio, reference_audio (+ optional reference_audio_2/_3 for multi-speaker mode), separation/detection/transcription/Fish S2 controls, time_match_mode, bg_extend_mode.

Outputs: final_audio, new_vocals_only, bg_audio_only, speaker_2_vocals_only, speaker_3_vocals_only, timeline_data.

Known limitations

• Demucs separation quality on non-musical ambience/foley (vs. the music stems it's trained on) is unproven beyond ear-testing on real clips.
• Segment detection is an energy threshold, not a trained VAD — expect some boundary hand-correction via the timeline on early runs.
• Auto speaker-assignment is a best-guess classifier; ambiguous segments are left unchanged rather than forced, by design — use the timeline to correct.
• Fish Audio S2 has no native duration/speed control, so ripple mode (the default) can drift the final duration away from the source over many segments — this is surfaced in the console log, not hidden.