Whisper Segments to Speaker

Whisper is great at turning speech into text and terrible at telling you who said it. That's not its job - it just transcribes. This node is the fix: it takes the segments Whisper produced, plus the speaker segments from the pack's Speaker Diarization node, and hands back the same transcript with a speaker label on every line. Same words, now a script instead of a wall of text.

If you're building the full pipeline, the shape is: audio → Speaker Diarization → Whisper Segments to Speaker ← Whisper transcription. It's the glue between two very different models, and it's the only thing in this pack you'll reach for after the diarization runs.

How it works

The logic is a pandas overlap match. It builds a table of every diarized turn (each with start, end, speaker), then for each Whisper segment it computes the intersection and union between that segment's time range and every speaker turn. Segments with positive overlap are kept, and the segment is assigned to the speaker whose turns overlap it the most - via groupby("speaker")["intersection"].sum().idxmax(). So a Whisper line that straddles a speaker change gets the person who owns more of it. If a segment overlaps nothing at all, it comes back as "Unknown".

There's nothing mysterious here, and that's a good thing. The code is small, readable, and does exactly what you'd hope: it never rewrites your transcript, just enriches it.

Inputs and outputs

Two required inputs:

• whisper_segments (whisper_alignment) - the segment list from a Whisper transcription node in your graph, where each segment carries start and end in seconds. The type name in ComfyUI is whisper_alignment; any Whisper node that outputs alignment segments will slot in.
• speaker_segments (speaker_segments) - the output of Speaker Diarization from this same pack.

The single output, segments_alignment, is the same whisper_alignment type you fed in, just with a speaker key added to each segment. Because the type is unchanged, it plugs straight back into whatever consumed the plain transcript - a text/print node, a subtitles writer, an LLM summary step. Nothing downstream needs to know the difference.

Installing it

This node ships in ComfyUI_pyannote, so you're installing the whole pack. Search that title in ComfyUI Manager, or:

cd ComfyUI/custom_nodes
git clone https://github.com/ramesh-x90/ComfyUI_pyannote
cd ComfyUI_pyannote
pip install -r requirements.txt

Restart ComfyUI and it appears in the node menu. Note that installing this pack pulls in pyannote.audio and torchaudio even if you only care about this node - the Whisper node itself is pure pandas, but you can't install it alone.

Where people get burned

Mismatched time units. The overlap math assumes both inputs are in seconds - Whisper's alignment and PyAnnote's diarization both use seconds, so the happy path is fine. If your Whisper node outputs milliseconds, every segment overlaps nothing and you get a transcript where everyone is "Unknown". Check the raw values once before you blame the node.

Every segment gets one speaker, even across a change. Because it picks the max-overlap speaker, a single Whisper segment that spans a handoff is labeled with just one person - the dominant one. If your transcript needs speaker turns to split at every change, you'll need finer-grained Whisper segments or a different step. That's a limitation of the approach, not a bug.

You need the diarization half working first. This node is useless without Speaker Diarization, and that node needs a gated Hugging Face model plus a valid token. Get that half running before you wonder why everything says Unknown.