ForgeAI Lyrics Transcriber

The ForgeAI-HeartMuLa pack gives you a generator that makes songs from lyrics. This is the node that takes the audio back to text - a Whisper-based transcriber that accepts a ComfyUI AUDIO input and hands you a lyrics string.

Why you'd want that: round-tripping. HeartMuLa doesn't always sing exactly what you typed, so generating a track, transcribing it, and seeing what it actually sang is the fastest way to fix your lyrics and regenerate. It's also handy for captioning a track you made elsewhere, or checking whether an existing song follows the [verse]/[chorus] structure you asked for.

How it works

The node is deliberately thin. It unwraps the ComfyUI AUDIO dict, writes the waveform to a temporary WAV file, resamples it to 16 kHz with librosa (Whisper's native rate - your source can be 48 kHz or anything else), and runs a Whisper transcription. It checks for a HeartMuLa-specific whisper model in ComfyUI/models/HeartMuLa/HeartMuLa-oss-whisper/, and if that folder isn't there - and it won't be, unless you put it there - it falls back to openai/whisper-large-v3. The temp file is cleaned up automatically.

One input, one output: audio in, lyrics out as a plain STRING you can feed back into the generator's lyrics field or save to a text node. It's not an output node, so wire it to something that displays or saves text.

Installation

Same pack, same install - this node ships alongside the generator, so:

cd ComfyUI/custom_nodes
git clone https://github.com/PavonicAI/ForgeAI-HeartMuLa.git
pip install -r ForgeAI-HeartMuLa/requirements.txt

then restart ComfyUI. No extra install steps for transcription itself.

The gotcha: a surprise 3 GB download

Here's where people get burned. The README's model layout lists the generator's model, HeartCodec, tokenizer.json, and gen_config.json - but not the whisper folder. So the first time you run this node, it downloads openai/whisper-large-v3 (~3 GB) from Hugging Face over the network. It works fine, but it's a long first run with no progress bar to look at, and it needs an internet connection. If you want to skip that, grab the HeartMuLa team's own transcription model and drop it in ComfyUI/models/HeartMuLa/HeartMuLa-oss-whisper/ - the node will use it instead.

Other things to know

• VRAM contention is real. whisper-large-v3 is a big model and it loads onto the GPU. If you run transcribe right after a long generation, you can hit memory pressure - the node does free the model after each call, so the next generation clears up.
• The resampling and temp-file dance happen on every call, so transcription is slower than a bare whisper CLI, but fine for a node that's doing one track at a time.
• It's Apache-2.0 like the rest of the pack, so no licensing surprises if you're using the output in a project.

For most people this node is a debugging companion to the generator rather than a daily driver. That's okay - when you're iterating on lyrics, having the transcription one node over in the same graph beats exporting the WAV and running Whisper by hand.