ComfyUI Audio Translator

ComfyUI custom node for:

• Speaker diarization with pyannote
• Transcription with Whisper
• Line-by-line translation with a local LLM (Qwen or compatible model in models/LLM)
• Optional dubbed audio synthesis with DramaBox, preserving speaker identity and timeline gaps

Node

• AudioTranslatorNode
• Category: audio

Inputs

• audio (AUDIO): source audio
• hf_token (STRING): Hugging Face token for pyannote models
• whisper_model (COMBO): tiny | base | small | medium | large | turbo
• merge_consecutive_speaker (BOOLEAN): merge neighboring same-speaker segments
• translation_language (COMBO): none, English, French, Spanish, ...
• llm_model (COMBO): model folder found under ComfyUI/models/LLM

Outputs

• dialogue_text (STRING): timestamped speaker dialogue in target language
• dubbed_audio (AUDIO): regenerated translated speech using DramaBox

How Dubbing Works

1. Pyannote diarizes speakers.
2. Whisper transcribes segments.
3. Each segment text is translated.
4. For each speaker, a reference voice clip is extracted from the original audio.
5. DramaBox generates each translated line with prompt pattern:
   • The person says with a <language> accent: "<translated text>"
6. Silence is inserted between generated lines when original timeline has gaps.

Dependencies

Install in ComfyUI embedded Python environment:

python -m pip install -r requirements.txt

requirements.txt:

• pyannote.audio
• openai-whisper
• transformers
• numpy
• imageio-ffmpeg

Also required for dubbing:

• ComfyUI-DramaBox custom node available at:
  • ComfyUI/custom_nodes/ComfyUI-DramaBox

Installation

1. Clone into ComfyUI/custom_nodes/Comfyui_audio_translator
2. Install dependencies
3. Restart ComfyUI

Performance Notes

• The node explicitly offloads translation/ASR/diarization models before DramaBox synthesis to reduce VRAM pressure.
• Dubbing is line-by-line; runtime depends on number of segments.
• merge_consecutive_speaker=true usually speeds up dubbing and improves flow.

Security

• Do not share workflow JSON with live hf_token values.
• If a token was exposed, rotate/revoke it immediately in Hugging Face settings.

Known Constraints

• Overlapping speech is not mixed as simultaneous voices; segments are emitted sequentially.
• Output prosody depends on reference quality and DramaBox generation behavior.