ITL Whisper Transcribe

Audio in, text out. That's the whole node, and it's the piece that makes the rest of this pack's audio suite actually usable: Breeze TTS voice cloning conditions on a (clip, exact transcript) pair, so a wrong transcript means a wrong voice. Rather than transcribe your reference clip in some other app and paste it across, you wire it here.

It's also a general transcription tool, and worth having for that alone. Subtitles for a generated video. A transcript of a spoken script so you can edit it as text. Whisper's output as a STRING goes into anything that takes text - a prompt box, a Save Text node, an LLM node downstream.

What you wire in

model comes from ITL Whisper Loader - the pack's own ITL_WHISPER type.

audio is any ComfyUI AUDIO. Note the tooltip: batch item 0 is transcribed. If you're feeding a multi-file loader, only the first clip in the batch gets read. Stereo is downmixed automatically, so you don't need to pre-mono anything, and any sample rate is accepted.

language is auto by default, or one of the 28 ISO 639-1 codes Whisper's tokenizer knows (en, zh, ja, de, fr, es, ru and so on). Leave it on auto almost always; pick a code when detection guesses wrong, which happens most on very short clips and on audio that switches language mid-sentence.

unload_after frees the Whisper model when the node finishes. Turn it on when VRAM is tight and you won't need transcription again in this run; the next Whisper node reloads it.

Output: a single text string.

How it works

Two conversions happen before the model sees anything. The waveform's channels are averaged into mono, and the sample rate is resampled to Whisper's fixed 16 kHz - that resample is librosa's job, which is why it's in the pack's requirements. Resampling only happens if your audio isn't already 16 kHz.

Then the length decides the path. Clips of 30 seconds or less go through Whisper's normal short-form decode, padded to its fixed mel-frame window. Anything longer is handed to transformers' sequential long-form decoding, which keeps every frame instead of truncating, carries an attention mask, and tracks timestamps across the whole clip. The node also disables conditioning on previous tokens - a small but important detail, because that's the setting that makes long-form Whisper loop on a repeated phrase. Your 20-minute narrated file comes back in one piece, not the first thirty seconds.

Finally the text is whitespace-normalized - newlines and odd spacing collapsed into single spaces. Punctuation and casing come from the model as-is.

Install

Ships with the pack; there's nothing extra to download here. Manager → search ComfyUI-IntoTheLatent-Utils → install → restart, or:

cd ComfyUI/custom_nodes
git clone https://github.com/Into-The-Latent/ComfyUI-IntoTheLatent-Utils
pip install -r ComfyUI-IntoTheLatent-Utils/requirements.txt

The first transcription triggers the checkpoint download (large-v3-turbo is 1.6 GB). Whisper weights are Apache 2.0 and run on CPU too - set the loader's device to cpu if you'd rather keep the GPU free.

Gotchas

Empty audio errors rather than returning an empty string, which is the behaviour you want: a silent failure here would quietly poison your reference text downstream.

Missing librosa gives you an install hint naming the requirements file. If resampling fails with an import error, that's the fix - not a node bug.

Chinese, specifically: Whisper emits either simplified or traditional characters depending on the audio it hears. If you're using the result as a Breeze reference_text, look at it before you commit - a transcript in the wrong script isn't an exact transcript.

The text is only as good as the audio. Whisper is excellent on clean single-speaker speech and struggles on music, overlapping voices and heavy noise. For the clone/direction workflow, use a clean reference clip: a few seconds, one speaker, no background music. That's good practice for the clone anyway, and it's what gives Whisper an easy job.

unload_after really does free it - the node is written so no lingering reference to the model survives the call, so the ~2 GiB (turbo, fp16) comes back rather than being pinned by the graph's cache. Which, if you've read the rest of this pack's audio nodes, is the recurring theme: what's resident is resident until you say otherwise.