Audio Translator (Pyannote+Whisper+Qwen)

Audio Translator is a whole pipeline wearing a single node's trench coat. Feed it any audio and it works out who's talking, writes down what they said with timestamps, translates every line into a target language with a local LLM, and - if you want the full party trick - regenerates the speech in each speaker's voice via DramaBox TTS. Transcript and dubbed track come out the other side. No cloud, no API key, nothing leaves your machine except your GPU's tears.

That last part is both the appeal and the catch. This is the kind of task that usually means stitching together three custom packs, a Whisper install, and a prayer. Here it's one node. But "one node" means one big pile of models, and the first run will download them all.

How it actually works

The pipeline is four stages, and the node is upfront about them in its own code:

1. Diarization. Pyannote's speaker-diarization-3.1 splits the audio into speaker turns - "Speaker A" through however many voices it finds.
2. Transcription. OpenAI Whisper transcribes the whole clip, then each Whisper segment is assigned to the speaker it overlaps most with. You get lines like 1:23 Speaker A: ....
3. Translation. Each line goes to a local LLM from your ComfyUI/models/LLM folder (Qwen or any compatible causal model; it'll even load a vision-language model if that's what you have). Greedy decoding, temperature zero - it's there to translate, not to improvise.
4. Dubbing. For each speaker it cuts a 3–12 second reference clip from the original audio, then DramaBox re-synthesizes every translated line with the prompt The person says with a <language> accent: "<translated text>". Silence is padded back in where the original had gaps.

One thing the author got right: the node offloads the diarization/ASR/LLM models from VRAM before the TTS stage kicks in. Running pyannote + Whisper + Qwen + a TTS at once would blow up even a 24GB card, so it staggers them.

The inputs that actually matter

• audio (AUDIO) - wire in anything that emits ComfyUI audio, e.g. a VHS or Load Audio node. This is the only wiring you need to do.
• hf_token (STRING) - your Hugging Face token. Required. Pyannote's diarization model is gated: you have to accept the terms on its model card and pass a token or the node errors out on auth. No way around it.
• whisper_model - tiny / base / small / medium / large / turbo. small is the default and a sane starting point; large/turbo are ~1.5GB downloads and slow on CPU.
• translation_language - 14 options plus none. Set none and the node behaves as pure speaker-diarized transcription, no LLM, no dubbing of translated lines (DramaBox still runs - see below).
• merge_consecutive_speaker (BOOLEAN, default on) - glues neighboring same-speaker segments into one line. Leave it on; it also speeds up dubbing, since DramaBox generates per line.
• llm_model - auto-populated from folders containing a config.json under ComfyUI/models/LLM. If that folder is empty you'll see the placeholder "Download any LLM to the models_LLM folder".

Outputs

• dialogue_text (STRING) - the timestamped M:SS Speaker A: text script, translated if you asked. Wire it to a Preview Text / Show Text node (it's also printed to the console log).
• dubbed_audio (AUDIO) - the regenerated translated speech, ready to go into a Save Audio node and mix under the original video.

Install

Easiest via ComfyUI Manager (search "comfyui_audio_translator"). Or the manual route:

cd ComfyUI/custom_nodes
git clone https://github.com/pmarmotte2/comfyui_audio_translator
cd comfyui_audio_translator
# inside ComfyUI's embedded Python
python -m pip install -r requirements.txt

Then restart ComfyUI. The requirements (pyannote.audio, openai-whisper, transformers, numpy, imageio-ffmpeg) are heavy, so expect a long first install.

You also need ComfyUI-DramaBox (git clone https://github.com/FranckyB/ComfyUI-DramaBox into custom_nodes) plus its own model downloads, a Qwen/LLM in models/LLM (create the folder if it doesn't exist), and the HF token above.

Where people get burned

• DramaBox isn't really optional. The README calls it optional, but the shipped code loads it on every run - even with translation off. If it's missing, the node fails at the end after all that transcription work. Install it before you run.
• Pyannote gating. "401 unauthorized" or model-not-found almost always means you haven't accepted the model terms or the token is wrong.
• VRAM. On 8GB cards use tiny/base Whisper and a small LLM; the node's offloading helps but this is a big pipeline.
• Don't share the workflow JSON with your hf_token baked into it - it's a live credential. If one leaks, rotate it on Hugging Face.
• DramaBox prosody is a coin flip. It's expressive and hallucination-prone by reputation; the dubs are best-effort, and quality tracks how clean your reference clips are.