IF Whisper Speech🌬️

IF Whisper Speech🌬️ is the audio half of the classic "LLM writes a script, then someone has to speak it" problem. You paste text in, pick a voice, and it comes out as speech that actually sounds like that voice - cloned from a single audio file, no LoRA, no training job, no API key. It's one node from ImpactFrames-YT's IF_AI pack family (same author as the IF_AI_tools LLM nodes), and it wraps Collabora's open-source WhisperSpeech model. The name is a little confusing - it has nothing to do with OpenAI's Whisper transcription. WhisperSpeech just borrows Whisper's encoder backbone for a text-to-speech model.

The author's own pitch is a talking-avatar workflow: feed an image plus a question to an IF_AI LLM node, pipe the answer's text in here, pick a voice, and you get narration for your video. On his 3090, roughly 20 seconds of speech took under a minute.

How it works

WhisperSpeech is a two-stage pipeline. A t2s model turns your text into semantic tokens, a s2a model turns those into audio code tokens conditioned on a speaker embedding, and a vocoder decodes the result into a waveform. The node does the orchestration around that: it splits your text into sentences (nltk), merges them into chunks under ~20 seconds, generates each chunk, and stitches them with an overlap window so long passages don't have seams - it even pads half a second of silence between chunks.

Voice cloning is the "on the fly" part. Drop a .ogg or .wav into ComfyUI-IF_AI_WishperSpeechNode/whisperspeech/audio/, restart ComfyUI, and it appears in the speaker dropdown. Select it and the node extracts a speaker embedding from that clip and conditions the s2a stage on it. The repo ships one sample voice (Pigston_Banker_ill.ogg) so you can hear it work before adding your own. The author recommends a clean, isolated voice track at 16kHz or lower sample rate for the best clone.

First run downloads the model weights from HuggingFace (collabora/whisperspeech) and pulls nltk's punkt_tab tokenizer data at import time, so make sure you have network access on the first launch.

The inputs that matter

• text - your script. Multiline. Right-click can convert upstream text input (like an LLM node's output) into this, which is how the "turn a chat into a book" workflow works.
• speaker - the dropdown of voice files from the audio folder, or None for the default voice.
• cps - characters per second, default 14, range 10–20. Lower it for a slower, more deliberate delivery.
• overlap - default 100, range 0–200. How much of the previous chunk's tokens carry over when stitching. Leave it alone unless you hear glitches at chunk boundaries.
• torch_compile - off by default. It can speed up inference, but the author himself ran without it, and torch.compile is exactly the kind of thing that misbehaves on Windows or older GPUs. Flip it on only if your setup handles it.

file_name just sets the output filename prefix.

Outputs

There are three, but you'll realistically use two. output_audio is the standard ComfyUI audio dict (waveform + sample_rate at 16kHz) - that's the one to wire into your video/audio mixing nodes. wav_16k_path is a string path to the saved .wav, which lives in ComfyUI/output/<file_name>_<timestamp>/; handy for passing to nodes that take a file path instead of an audio object. The first output, audios, is a raw list of decoded tensors rather than a proper audio dict - a quirk. If in doubt, use output_audio.

Install

Easiest via ComfyUI Manager - search "IF_AI_WishperSpeechNode". Or manually:

cd ComfyUI/custom_nodes
git clone https://github.com/if-ai/ComfyUI-IF_AI_WishperSpeechNode
cd ComfyUI-IF_AI_WishperSpeechNode
pip install -r requirements.txt

Then restart ComfyUI. This pulls real dependencies - WhisperSpeech, librosa, torchaudio, speechbrain among them - so on a fresh environment expect a chunky install. Windows portable users can run the included embedded_install.bat.

Gotchas

The README spends most of its ink on dlib install workarounds (cmake, conda-forge, admin terminals). Here's the thing: that's largely legacy from the parent IF_AI_tools pack - this repo's requirements.txt doesn't actually list dlib. But the author kept the fixes around because the speaker-embedding stack can still drag it in on Windows. If you hit a dlib error, follow the README: pip install cmake first, then pip install dlib, or conda install -c conda-forge dlib if you're on conda.

The other thing to expect: your voice file won't show up until a restart, and the clone is only as good as the clip you feed it. Muddy, overlapping audio gives a muddy voice. Feed it clean speech and it's genuinely impressive for a single node - and one of the friendlier ways to get a talking-head pipeline's audio sorted without leaving ComfyUI.