Process Whisper Diarization (mtb)

Whisper gives you what was said. This node adds who said it. Diarization is the task of splitting audio by speaker - Speaker 1 said this, Speaker 2 said that - and this node takes your Whisper transcription chunks plus the original audio and labels each chunk with a speaker. If you're transcribing an interview, a podcast, or any back-and-forth dialogue, this is what turns a wall of text into a properly attributed conversation.

It's the most advanced node in MTB's audio set, and also the most demanding to set up, so go in knowing that. When it works, it's the difference between "here's a transcript" and "here's a transcript that reads like a script."

How it works

It runs a speaker-diarization model over the audio to figure out the speaker turns, then aligns those turns with the timestamped chunks from your Whisper transcription so each piece of text gets a speaker tag. You pick which diarization engine does the work.

The inputs that matter

• whisper_chunks - the timestamped transcription chunks (WHISPER_CHUNKS) coming from the Whisper transcription step. Diarization needs the timing of the words to line speakers up with text.
• audio - the original audio, which the diarization model listens to directly (it works from the sound, not the text).
• backend - pyannote (default) or nemo. Pyannote (pyannote.audio) is the widely-used standard; NeMo is NVIDIA's toolkit. They're two different engines for the same job - start with pyannote unless you have a reason to prefer NeMo.
• num_speakers (default 2) - how many distinct speakers to expect. Set this to the real number if you know it; a good count dramatically improves the split.
• device (optional, cuda/cpu) - run on GPU for anything but the shortest clips.

The single output is WHISPER_CHUNKS - the same chunks, now carrying speaker labels, ready for a subtitle or export node.

How to install it

ComfyUI Manager: search MTB Nodes (comfy_mtb), install, restart. Manually: cd ComfyUI/custom_nodes && git clone https://github.com/melMass/comfy_mtb, then restart. This is the MTB audio node most likely to not load on a fresh install, because the diarization backends are heavy optional dependencies - pyannote.audio for the pyannote path, the NeMo toolkit for the nemo path. If the node's missing, MTB's dependency panel at http://127.0.0.1:8188/mtb is where you install them.

Common issues

The setup gotcha that catches everyone using pyannote: its models are gated on Hugging Face. You have to visit the model page, accept the conditions, and provide an access token before it will download - otherwise you get an authorization error that looks like a bug but isn't. Do that once and it works. NeMo avoids the gating but pulls in a large toolkit of its own.

Beyond that, num_speakers is the accuracy lever: guess it wrong - say 2 when three people are talking - and the labeling gets muddled, with one real speaker split across two labels or two merged into one. Set it to the true count. And run on cuda; diarization on CPU is painfully slow on anything longer than a short clip. This node rewards a bit of setup patience - it's fiddly to stand up, and genuinely useful once it's running.