Whisper Node

Whisper Node is the front half of the Schedulizer pipeline: it takes an audio file and returns the lyrics as plain text plus a version with per-line timestamps. On its own that's just speech-to-text - handy if you want subtitles or captions out of ComfyUI without leaving the graph. The reason it exists is the second output, which feeds the pack's Prompt Schedule Converter to build prompt-travel schedules for music videos. If you've ever wanted your AnimateDiff render to change its prompt in sync with a song's lyrics, this is the part that gets the lyrics out of the audio.

How it works

It's a thin wrapper around OpenAI's local Whisper - no API, no upload, no key. It takes an AUDIO input (from ComfyUI's built-in LoadAudio or any audio-producing node), writes it to a temp .wav via torchaudio, runs whisper.load_model(model).transcribe(...), and formats the result. The plain text output is the full transcription; text_with_timestamps is one line per segment in the [0.00s - 5.32s] lyric format.

Two mechanism details worth knowing before you queue a long render:

• Models download on first use. The first time you pick a model, Whisper pulls it into ~/.cache/whisper. large/large-v2 are roughly 3 GB; base is about 142 MB. The workflow that ships with the pack defaults to large-v2, so the very first run can look like a hang while it downloads.
• It transcribes the whole file every time you run the graph - and reloads the model on each execution. There's no caching of results. If you're iterating on the rest of the workflow, every queue re-transcribes from scratch, which is slow on long tracks.

Also note it passes word_timestamps=True but builds its lines from Whisper's segment timestamps, not words. In practice a "segment" is a sentence, so your schedule flips per line, not per word. That's what you want for lyric-synced prompts anyway.

The inputs you'll set

• audio (AUDIO): wire in your song or clip. The pack's example workflow uses LoadAudio → Whisper Node, and that's the obvious start.
• model (enum: base, tiny, small, medium, large, large-v2): accuracy vs. speed. Whisper is trained on speech, and song lyrics with music behind them are hard - on a busy mix the small models will mangle lines and give you bad prompts downstream. Start with medium or large-v2 if the track is anything but clean vocals; drop to small only for quick tests. The list tops out at large-v2 - no turbo, no large-v3, so what you see is what you get.

Outputs are text (plain transcription) and text_with_timestamps (the timestamped lines). The latter plugs straight into the Prompt Schedule Converter's text_with_timestamps input - the regex there matches this node's format exactly, because the pack is designed as one pipeline.

Installing it

Install the pack once and both nodes appear. ComfyUI Manager: search "Schedulizer", install, restart. Or manually:

cd ComfyUI/custom_nodes
git clone https://github.com/DoctorDiffusion/ComfyUI-Schedulizer.git
cd ComfyUI-Schedulizer
pip install -r requirements.txt

This is the node that drags in the heavy dependency: requirements.txt lists openai-whisper (which pulls in PyTorch bits if your environment somehow doesn't have them), plus pillow and uuid - uuid is a Python stdlib module, so that entry is harmless fluff. torchaudio gets imported at runtime but isn't in the requirements; in practice it ships with the torch that ComfyUI already installs, so it works out. If you ever see an ImportError on torchaudio, that's what's going on.

Common issues

• First run appears frozen - that's the multi-GB model downloading to ~/.cache/whisper. Give it a minute; it's one-time per model.
• Bad lyrics on music - Whisper chokes on dense mixes. Use the larger models, and proofread before you build a whole prompt schedule off the transcript, because garbage in here propagates straight into your prompt travel.
• No audio - make sure your source node actually outputs AUDIO type. LoadAudio works; a path string does not.

The pack is small and hasn't been touched since late 2024, but it wraps two stable libraries and the pipeline still holds up. For the specific job of turning a song into a frame-synced prompt schedule, it's the least-friction path in ComfyUI.