Multi-Speaker Speech To Text (ASA)

Multi-Speaker Speech To Text (ASA) answers the question the rest of this pack assumes you already have an answer to: given a recording of two or more people talking, who said what, and when? It takes an AUDIO clip, figures out how many speakers there are and where each one talks (that's diarization), transcribes every segment, and hands you the whole thing as one JSON string. And that JSON is the exact input Character Vocal Extractor is built to consume.

This is the node that makes the pack's dubbing workflow work without hand-annotating timestamps. Point it at your source recording, get the speaker-tagged transcript, feed that into the extractor, and you can pull each person's voice onto its own track - re-record one actor, re-voice one character, clean one speaker out of a noisy interview. On the English-speaking internet this pack barely has a footprint, so know that going in: the docs are Chinese-first, and the STT machinery is the fiddliest part of the install. It's also genuinely the most capable node here if you get it running.

How it works

The audio is saved to a temp WAV, then run through pyannote.audio's speaker-diarization 2.1 pipeline to produce speaker-labeled time segments. Each segment is then cropped out and transcribed separately with faster-whisper using whatever whisper_model size you picked, with beam_size=5. The output is a JSON array where every entry has an id (the speaker label), start, end, and text:

[
  {
    "id": "SPEAKER_00",
    "start": "0:05",
    "end": "0:11",
    "text": "I think we should ship it anyway."
  }
]

The inputs that matter

• audio - the clip. The pack README recommends 16 kHz mono for the best diarization, and it's good advice.
• language - auto (default) or zh/en/ja/fr/de/ru/es/it/ko. Setting it speeds things up and improves accuracy.
• whisper_model - tiny, base, small, medium (default), or large-v2. Bigger is more accurate and much slower; on CPU, small is the sane choice.
• use_whisperx - leave it off. WhisperX is disabled in this build; the toggle currently falls back to pyannote anyway.
• auth_token - your Hugging Face token. Needed because pyannote's diarization model is gated.

Output: subtitle_json, a STRING you paste into the extractor's subtitle_json input.

Installing it - the extra step

The pack's requirements.txt does not include pyannote.audio. You must install it manually:

pip install pyannote.audio

Then get a Hugging Face token and accept the terms for the gated model pyannote/speaker-diarization on the HF site, and drop the token in the auth_token field. First run downloads the diarization model (into ComfyUI/models/speaker-diarization if it can) and the whisper model - those are multi-GB on the large side, so the first run is slow regardless of your hardware.

The rest is the usual: ComfyUI Manager → search "AudioSuiteAdvanced", or clone https://github.com/whmc76/ComfyUI-AudioSuiteAdvanced into custom_nodes and pip install -r requirements.txt.

Where people get burned

Three things, in order of pain. The token. Without it, diarization fails and the node returns an empty list - the console prints a pointer to the HF page, but the node itself just silently gives you []. Second, speaker IDs don't match the extractor's. pyannote emits SPEAKER_00, SPEAKER_01, etc., but Character Vocal Extractor looks for Narrator, Character1…Character5. You'll need a mapping step (any string-replace text node will do) before the extractor will see your speakers. Third, model default. medium is heavy; on a mid GPU or any CPU it's minutes per clip. Drop to small for iteration and go up only for the final pass.

One more: this node forces a re-run when its inputs change (it hashes the audio shape and params), so it won't silently serve you a stale transcript - but it also means long clips re-diarize on every parameter tweak. Iterate on a short test clip first.