Speaker Diarization (pyannote)

The pack is called ComfyUI-Speaker-Isolation, so you'd be forgiven for expecting separate audio files at the end, one per voice. That's not what this node does. What PyannoteDiarizationNode (display name "Speaker Diarization (pyannote)") actually gives you is a timestamped transcript where every line is tagged with a speaker: 0:10 SPEAKER A: ..., 0:20 SPEAKER B: .... Who-said-what-when, as text, straight out of the graph. Useful - but a common source of a "wait, this is all it does?" moment.

It's an odd bird in the ComfyUI ecosystem. Most custom nodes are image and video machinery, but this one sits in the audio category and leans on two heavyweight pieces of the speech-AI world: the pyannote/speaker-diarization-3.1 model from Hugging Face and OpenAI's Whisper. If you're building a podcast- or interview-processing workflow, generating speaker-labeled subtitles, or prepping audio datasets for a TTS or voice-cloning project, this is the rare node that does the whole two-step in one box instead of making you shell out to a Python script.

How it works

The mechanism is a clean two-stage pipeline. On input, it takes the ComfyUI AUDIO dict (waveform + sample rate - the kind of thing a video/audio loader node hands you), converts it to mono, normalizes it, and writes a temporary WAV. Then:

1. It loads the pyannote diarization pipeline with your HF token and runs it over that WAV, producing "speaker turns" - time ranges labeled SPEAKER_00, SPEAKER_01, ...
2. It loads a Whisper model (size from the dropdown) and transcribes the same WAV.
3. For each Whisper segment it picks the speaker whose turn overlaps the segment the most, then formats the whole thing as timestamped lines.

The details are where it gets smart. Segments with a high no_speech_prob, a low average log-probability, or shorter than 0.35 seconds get dropped. If a segment barely overlaps any speaker turn, it's skipped rather than guessed. And when merge_consecutive_speaker is on (it is by default), consecutive lines from the same speaker get glued into one block so the output isn't a wall of SPEAKER A: ... SPEAKER A: ....

The inputs that matter

Four inputs, and you only really touch two of them.

• audio (AUDIO) - whatever clip you're analyzing, wired in from a loader node.
• hf_token (STRING) - the one that bites. Empty string → the node raises "HF token is required" and stops. This isn't optional flavor; see below.
• whisper_model - tiny, base, small, medium, large, turbo, default small. small is the right default: fast enough and decent accuracy. turbo is noticeably better but hungry; on a 8GB card you'll feel it.
• merge_consecutive_speaker (BOOLEAN) - leave it on unless you want every Whisper segment as its own line.

The output is a single diarization_text string, one line per turn, timestamps in M:SS. Wire it into any text node (a ShowText-style viewer, a save-to-file node, or string manipulation) - it doesn't auto-save anything.

Install

ComfyUI Manager should find it as ComfyUI-Speaker-Isolation (the pack is registry-ready). The manual route:

cd ComfyUI/custom_nodes
git clone https://github.com/pmarmotte2/ComfyUI-Speaker-Isolation
cd ComfyUI-Speaker-Isolation
pip install -r requirements.txt

Then restart ComfyUI. The dependencies are the real story: pyannote.audio>=3.1.0, openai-whisper, torch, numpy. Torch you already have; pyannote drags in torchaudio, speechbrain, pytorch-lightning and friends, so expect a chunky install. First run downloads the pyannote pipeline and your Whisper model, so the very first execution is slow and needs network.

Common issues

The token is the number-one failure point. pyannote/speaker-diarization-3.1 is a gated model: you must log into Hugging Face, accept its license on the model page, then create an access token with read access to gated repos. A token you generated before accepting the gate won't work, and an empty field raises immediately.

The author also coded around two real environment landmines: PyTorch 2.6+ defaults torch.load to weights_only=True, which breaks older pyannote checkpoints (there's an allowlist in the code), and there's a workaround for a pytorch-lightning → inspect.stack() recursion that hits in some embedded setups. If you ever see recursion errors or "weights only load failed" from this or similar nodes, that's the terrain you're in.

Quality-wise, keep expectations realistic: pyannote is excellent but not magic on overlapping speakers, and any Whisper segment that straddles two voices gets assigned to whichever speaker overlaps most. Heavy crosstalk means a transcript that's right about what was said but loose about who said it. And again, despite the pack name - no isolated audio tracks come out. For labeled transcripts, this is one of the few ComfyUI-native ways to get them.