AIIA Generate Speaker Segments

Before you can label a transcript with "who said what", you need to know when each speaker talks. That's the job of AIIA Generate Speaker Segments: it runs NVIDIA's NeMo Sortformer end-to-end diarization model on a plain audio track and returns a structured set of time segments, each tagged SPEAKER_00, SPEAKER_01, and so on. Think of it as the "who's talking at this moment" detector that everything else in the diarization flow builds on. On a 4090 it chews through 10 minutes of audio in about 2 seconds.

How it works

Feed it an audio tensor. The Sortformer model analyzes the whole clip end-to-end (that's the "E2E" - one pass, no separate voice-embedding + clustering stages) and emits speech segments with speaker labels. Then the results are run through a post-processing step that merges or trims fragments based on the profile you pick. The single output, speaker_segments, comes out in the pack's WHISPER_CHUNKS-compatible format - so it can plug into AIIA E2E Speaker Diarization, which attaches those labels to your Whisper text chunks. Run together, they produce a fully speaker-tagged transcript.

The inputs that matter

• e2e_backend_model - the .nemo model file, and the thing that bites. When no model is on disk the dropdown reads NO_MODELS_FOUND and the node is dead in the water. Put a Sortformer .nemo file in ComfyUI/models/nemo_models/:

  cd ComfyUI/models
  mkdir -p nemo_models
  hf download nvidia/nemo-models diar_sortformer_4spk-v1.nemo --local-dir nemo_models

• num_speakers_hint - 0 lets the model auto-estimate (capped at 4, since Sortformer tops out there); give it an explicit count when auto-estimation misses a quiet participant.
• postprocessing_profile - the one you'll actually fiddle with. It's a presets ladder from very_permissive (keeps short, low-confidence speech fragments - good for noisy recordings) to very_strict (drops anything that isn't clearly speech). balanced is the default and right for clean podcast audio. If you keep losing quiet interjections, step down toward permissive.

The custom_* inputs only appear when you pick the custom profile: custom_onset/custom_offset are VAD thresholds (when speech "starts" and "stops" relative to volume), custom_min_duration_on/off filter tiny speech and silence blips, and custom_pad_onset/custom_pad_offset add breathing room around segments so you don't clip the first syllable.

Install & troubleshooting

Standard pack install: havvk/ComfyUI_AIIA via ComfyUI Manager or git clone https://github.com/havvk/ComfyUI_AIIA.git into custom_nodes/, restart. The model is the dependency that actually matters - no .nemo file, no work. If you hit a diarize() crash on modern PyTorch, that's the known lhotse incompatibility; the pack ships a patch for PyTorch 2.10+, so update the pack instead of pinning your torch version down. And if the labels come back with one speaker absorbing everyone, your audio probably has cross-talk - try num_speakers_hint set explicitly, or a cleaner source file.