XB-BOX - 🎤 CosyVoice3 零样本克隆

This is the node that makes CosyVoice3 feel like magic: give it a few seconds of someone talking, type what you want them to say, and it speaks your line in their voice. No fine-tuning, no LoRA, no training data beyond that one clip. It's the classic zero-shot cloning setup, and the reason people reach for CosyVoice3 over Chatterbox or F5 is speaker similarity - the clone is just closer to the original person.

It lives in the XB_ToolBox CosyVoice3 suite, which is worth calling out because for over a year the CosyVoice nodes in ComfyUI were written for CosyVoice1 and abandoned. This pack (WJLUOXIAO's XB-BOX) is one of the first to wire up the v3 model properly, and this node is its flagship.

How it works

Feed it three things: the model from XB_CosyVoice3_ModelLoader, the text you want synthesized, and a reference_audio clip of the voice you're cloning.

The clever bit is that you don't have to transcribe the reference audio yourself. The node runs the clip through Whisper to get the transcript, then formats it the way CosyVoice3 wants - with the You are a helpful assistant.<|endofprompt|> prefix that the v3 checkpoint requires. If transcription fails (or the audio is pure noise), it silently falls back to cross-lingual mode, which can clone a voice without any transcript at all. Both paths call the model's inference_zero_shot and stream out chunks that get concatenated into one waveform.

Reference audio is validated before anything runs: at least 0.5 seconds, at most 30, with 3–10 seconds the sweet spot. CosyVoice3 can work with as little as 3 seconds, and the community's launch reactions confirm it holds up there.

The inputs that matter

• reference_audio - your voice sample. A clean, single-speaker clip of 3–10 seconds. No transcript needed (Whisper handles it).
• text - what the cloned voice should say. Multiline, so paragraphs are fine.
• speed - 0.5x to 2x, default 1.0. Cosmetic tweaking after you have a voice you like.
• seed - default 42, or -1 for random. CosyVoice isn't hugely seed-sensitive, but setting it makes outputs reproducible when you're iterating on a take.
• text_frontend - leave it on. Turn it off only if you're feeding CMU phonemes or special tags like <slow> directly.

The single output is audio (AUDIO), which wires straight into a PreviewAudio or SaveAudio node, or into a digital-human/lip-sync pipeline downstream.

Install and pitfalls

The pack installs via ComfyUI Manager (search XB_ToolBox) or git clone https://github.com/wjluoxiao/XB_ToolBox.git into custom_nodes, then restart. First run needs the model download (a few GB, from HuggingFace or ModelScope) plus the CosyVoice dependency stack - see the loader article; if you hit an ImportError run pip install -r requirements.txt from the pack folder.

Where people get burned: feeding a recording with music, room echo, or a second speaker in the background - the clone picks all of it up. Feed it music and you get singing. Also, the first time you use it the console will pause loading Whisper's base model; that's a one-time download, not a hang. And if you hear a brief click or pop at the start of longer outputs, that's the chunk concatenation seam, not a broken install - trim the clip edges in an audio editor if it bothers you.

One honest caveat: this runs at roughly 1.2–1.4x real time on a strong GPU. A 30-second line takes about 40 seconds to generate. Fast enough for iterating, not fast enough to treat like an API.