AIIA E2E Speaker Diarization

Whisper will happily transcribe a two-person recording, but it won't tell you which person said what. That's the gap this node fills: it takes your Whisper-style transcript chunks plus the original audio and stamps each chunk with a SPEAKER_00 / SPEAKER_01 label. End result - a subtitle file where you can color-code by character, or feed cleanly segmented lines into an isolation node.

How it works

Diarization here is a two-node story, and this is the second one:

1. AIIA Generate Speaker Segments runs NVIDIA's NeMo Sortformer E2E model on the raw audio and figures out who's talking when. Fast, too - the README claims ~2 seconds for 10 minutes of audio on a 4090.
2. AIIA E2E Speaker Diarization takes that segmentation plus your text-bearing whisper_chunks and assigns the best-matching speaker label to each transcript chunk.

So your inputs here are whisper_chunks (from whatever Whisper node produced the text) and audio. The output is a single thing: diarized_whisper_chunks, ready to hand downstream.

The inputs that matter

• backend_model - the Sortformer .nemo file. This dropdown defaults to NO_MODELS_FOUND when the model isn't on disk, which is the #1 wall people hit. You must put a .nemo file in ComfyUI/models/nemo_models/ - the README recommends nvidia/nemo-models diar_sortformer_4spk-v1.nemo (the streaming v2.1 variant also works):

  cd ComfyUI/models
  mkdir -p nemo_models
  hf download nvidia/nemo-models diar_sortformer_4spk-v1.nemo --local-dir nemo_models

• num_speakers - 0 lets the model auto-estimate; you can pin 1–4. Sortformer tops out at 4 speakers, so a crowded panel will confuse it - know your cast.
• device - cuda (default) or cpu. CPU works but "2 seconds on a 4090" becomes "2 minutes on a laptop", so only drop to CPU if you must.

Install & gotchas

The pack itself installs via ComfyUI Manager (search ComfyUI_AIIA) or git clone https://github.com/havvk/ComfyUI_AIIA.git into custom_nodes/, then restart. Beyond the missing-model trap, the other classic is a NeMo/PyTorch crash on the diarize() call - the pack has a compatibility patch for PyTorch 2.10+ that monkey-patches lhotse's CutSampler, so if you're on a recent PyTorch, keep the pack updated rather than fighting it. If labels come back jumbled, your audio may have a lot of cross-talk or music; feeding a cleaner, louder-normalized file and nudging num_speakers usually sorts it.

One note: whisper chunks are expected in the pack's WHISPER_CHUNKS format. If your transcription node outputs plain text or a different structure, this node won't take it - generate segments first and use that structured output.