Nodes/Kinburg-Nodes/Echo (Lyrics โ†’ Timing) ๐Ÿ’ฌ
ComfyUI Node

Echo (Lyrics โ†’ Timing) ๐Ÿ’ฌ

Echo already knows the words โ€” it just finds out when they were sung

By KinburgยทCreated 3 months agoยทUpdated 2 days agoยท 1
Echo (Lyrics โ†’ Timing) ๐Ÿ’ฌ
  • audio
  • vocals
  • voice_1
  • voice_2
  • voice_3
  • voice_4
  • timing
  • plan
  • ass_path
  • srt_path
  • report
โ—„lyricsโ€”โ–บ
โ—„planโ€”โ–บ
โ—„languageautoโ–บ
โ—„slack0.15โ–บ
โ—„write_asstrueโ–บ
โ—„write_srtfalseโ–บ
โ—„filename_prefixclips/echoโ–บ
โ—„colorsโ–บ
โ—„fontArialโ–บ
โ—„font_size0โ–บ
โ—„frame_width1920โ–บ
โ—„frame_height1080โ–บ
โ—„karaoke_sweeptrueโ–บ
โ—„lead_in0.35โ–บ
โ—„deviceautoโ–บ
โ—„halftrueโ–บ
โ—„unload_afterfalseโ–บ
โ—„verbosetrueโ–บ
โ—„anchortrueโ–บ

You've generated a song locally, the lyrics are sitting on disk as text, and you want two things: the words on screen in time with the voice, and your cuts landing where the chorus actually starts. Echo does both, because they're the same problem.

Alignment, not transcription

The obvious move is Whisper - what was sung, and when. That's the harder question. You already know the words, so asking a model to recognise text you can verify just gets you approximate lyrics plus timings that are an estimate of an estimate. Echo inverts it: this is forced alignment, where the words are given and the model only answers where each one sits. It cannot invent a word or drop one, which is why it works on a full mix with no separation step and why the result feels oddly exact next to an ASR pass. It's the middle of Kinburg's music-video chain (Siren sings, Echo times, Orpheus cuts, Save Clip burns), but it takes an mp3 and a text file just fine.

How it works

The model is MMS_FA, Meta's wav2vec2 fine-tuned for alignment across 1130 languages, about 1.2 GB. torchaudio's forced_align walks a Viterbi path through a CTC lattice and hands back a confidence per word - the number you judge a run by.

Attention over 20 ms frames is quadratic in what you hand it, so the song is never handed over whole: it's cut into one block per plan row, each encoded on its own, 15โ€“45 s at a time. Peak memory is set by your longest section, not the song's length.

anchor (on by default) is the clever bit. Each section is searched from where the previous section's words actually stopped, not where the plan says it begins. Plans drift progressively - one real take ran +1.5 s behind at the first verse and +17.7 s by the second - so the last measurement is the next section's best correction. And since a section can't begin before the previous one stopped singing, a section that was never sung gets searched where it belongs, finds nothing, and says so, rather than landing confidently on top of whoever is singing there. Turn it off to read every window straight off the plan again.

Inputs and outputs

Required are just two: audio and lyrics.

  • audio is the song and it's the clock - its length is what the plan's sections scale onto, so it must be the whole track. Never put an instrumental stem here; the timings will look confident and be nonsense.
  • lyrics is the same sheet that went to Siren, markers and all.
  • vocals (optional) is the isolated vocal that actually gets listened to. It won't fix a section searched in the wrong window - that's arithmetic about the plan, not hearing - but it lifts the confidence column, which is the difference between trusting a placement and only suspecting one. Must be exactly as long as audio.
  • plan bounds each search window and carries the voice column. Leave it empty and the lyric is spread over the song by syllable count - slower and less certain.

Then slack (how far outside its planned span a section may be found) and voice_1โ€ฆvoice_4 with colors (a colour per singer with no diarization at all - the plan carries the voice, and [Chorus - Nina] resolves to a card).

Outputs: timing (into Save Clip's subtitles, or Orpheus), plan - rewritten so every section sits where its words really are, the same table shape the downstream nodes read - ass_path, srt_path, and report, which prints per-section drift beside confidence. That report is how you tell a section that genuinely moved from one the aligner lost.

Install

Via ComfyUI Manager (search "Kinburg"), or:

cd ComfyUI/custom_nodes
git clone https://github.com/Kinburg/Kinburg-Nodes
# restart ComfyUI

Echo needs nothing extra - torchaudio ships with ComfyUI. The 1.2 GB MMS_FA weights download on first run into ComfyUI/models/stt/echo/, not into a hidden ~/.cache, so you can find them and delete them later.

Where it goes wrong

Instrumental in the audio slot. The number one way to waste an hour.

Drift against the window edge. Forced alignment must place every word in the window it's given, so a section that really starts 15 s late gets crammed against the far edge - appearing early, over whoever's singing, while the real performance passes unsubtitled. If the report shows that across the song, raise slack. The cost is encode time.

A line the model never sang. AceStep swallows words and repeats the last phrase on a long tail. Echo lays those words down regardless and low confidence is the only tell - read the weakest lines, not the average.

VRAM. device = cpu works, roughly ten times slower. unload_after frees the 1.2 GB when the graph goes straight into a video model. The .ass font is resolved by your player, not by Echo, so font must exist on whatever opens the file.

Check the timing before rendering anything: with write_ass on, open the song and the .ass in any player and the words light up in seconds, no render at all - and it restyles afterwards without re-encoding. write_srt adds a plain SubRip for anything that can't read ASS.

CategoryKinburg-Nodes/Bestiary/Echo

Inputs (25)

NameTypeDefaultDescription
audioAUDIOThe song โ€” straight out of Siren, or a LoadAudio for something off disk. Nothing needs to be separated first: the aligner is looking for words it already knows, not trying to hear them out of nowhere. This input is the CLOCK. Its length is what the plan's sections are scaled onto, so it has to be the whole track, start to end. NEVER put the instrumental stem here. It is also what gets listened to whenever 'vocals' is absent or rejected, and aligning a lyric against music with no voice in it produces timings that look confident and are nonsense. If a separated vocal is all you want to wire, put it HERE and leave 'vocals' empty โ€” a full-length stem is its own clock and the result is identical.
lyricsSTRINGThe SAME lyric sheet that went to Siren, markers and all. This is the whole reason the node is cheap: the words are known, so the model is only asked WHERE they are, never what they were. A forced alignment cannot invent a word or drop one.
vocalsoptAUDIOThe isolated vocal, if you have one โ€” from 'Vocals using MDX' or any other separator. This is what gets LISTENED to; 'audio' stays the clock and the fallback. Wiring both is the belt-and-braces arrangement, and the reason to prefer it over putting the stem in 'audio' alone: a separator sometimes swallows a quiet or whispered passage entirely, and when that happens the mix is still there to be the clock and to take over. What it buys: the acoustic model was trained on speech, so a dense arrangement lowers its confidence on every word, worst on consonants. An isolated vocal raises the whole confidence column โ€” which matters less for PLACING a word than for TRUSTING the placement. A weak reading and a wrong one both look like 0.3; at 0.8 against 0.3 they are telling you different things. What it does NOT fix: a section searched in the wrong window. That is arithmetic about the plan, not hearing, and no separation changes it. It must be the same length as 'audio' โ€” a separator that trimmed silence would shift every timing by however much it cut, so a mismatch is refused rather than quietly used.
planoptSTRINGSiren Score's or Siren Cast's plan. Two jobs: โ€ข it says roughly where each section is, which is what bounds the search window โ€” and the window is what keeps a three-minute song from costing three minutes of quadratic attention; โ€ข it carries the VOICE column, which is where per-singer colour comes from without any speaker diarization at all. Leave it empty and the lyric is spread over the song by syllable count with much wider windows. That works, and it is several times slower and less certain.
voice_1optKINBURG_VOICEThe band, as Character Card's 'voice' output โ€” the same cards wired into Siren Score. Used to resolve a marker like '[Chorus - Nina]' to a member, so a solo line inside a duet chorus gets Nina's colour rather than the whole duet's. With nothing wired the plan's own voice cell is used, which is usually right and cannot tell two singers apart inside one section.
voice_2optKINBURG_VOICEโ€”
voice_3optKINBURG_VOICEโ€”
voice_4optKINBURG_VOICEโ€”
languageoptCOMBOautoWhich spelling rules turn the lyric into the aligner's 28-letter alphabet. 'auto' decides PER WORD from the letters that exist in only one language (ั— ั” า‘ against ั‹ ั ัŠ ั‘), falling back to whatever the lyric is mostly in โ€” so a line with a Ukrainian and a Russian word in it is spelled correctly on both. Set it explicitly only if a song is in one language and 'auto' is guessing badly.
slackoptFLOAT0.150โ€“1How far outside its planned span a section is allowed to be found, as a fraction of its own length (never less than 3 s). This is the dial for the author's actual complaint โ€” that the plan and the performance disagree. Raise it when the report shows sections drifting up against their window's edge; lowering it makes the search stricter and faster, and risks pinning a section to a place it is not.
write_assoptBOOLEANtrueWrite a .ass subtitle file next to the video folder. This is the FAST way to check the timing: open the song and the .ass in any player and the words light up as they are sung, in seconds, with no render at all. It is also restylable afterwards without re-encoding anything.
write_srtoptBOOLEANfalseAlso write a plain .srt โ€” one cue per line, no karaoke. For anything that cannot read ASS.
filename_prefixoptSTRINGclips/echoSave path prefix under ComfyUI/output, with a counter appended โ€” the same convention Save Clip and Save Song use.
colorsoptSTRINGOne 'Name = #rrggbb' per line, e.g. 'Nina = #ff66cc'. Any singer not named here takes the next colour off a built-in palette, in order of first appearance โ€” so this can stay empty and still give every voice its own colour. The names are matched to the plan's voice column and to the cards, case- and space-insensitively.
fontoptSTRINGArialFont name for the .ass file. It is resolved by the PLAYER, not here, so it has to be a font installed on whatever plays the file. Arial and Segoe UI both carry Cyrillic.
font_sizeoptINT00โ€“4000 = scale it to the frame (about 1/22 of the short side), which is what keeps a subtitle readable on both a 16:9 and a 9:16 clip. Anything else is used verbatim.
frame_widthoptINT192016โ€“8192The frame the .ass is laid out for. It only sets PlayRes โ€” the file scales to whatever it is played at โ€” but getting it right makes the default font size land correctly.
frame_heightoptINT108016โ€“8192โ€”
karaoke_sweepoptBOOLEANtrueThe highlight FILLS across each word as it is held (\kf) rather than flipping at its start (\k). Fill reads better on slow lines and worse on fast ones.
lead_inoptFLOAT0.350โ€“3Seconds a line appears BEFORE its first word is sung, so it can be read rather than only followed. Nothing is highlighted during the wait.
deviceoptCOMBOautoWhere the aligner runs. 'auto' is ComfyUI's own device. CPU works and is perhaps ten times slower; use it if something else is holding the VRAM.
halfoptBOOLEANtrueRun the encoder in fp16 on CUDA โ€” half the weights and half the activation memory. The alignment itself is always done in fp32, so this costs nothing in precision that matters.
unload_afteroptBOOLEANfalseFree the 1.2 GB of weights when the node is done. Off by default because subtitles get re-styled several times per song and reloading each time is seconds wasted; turn it on when the same graph goes straight into a video model that wants every byte.
verboseoptBOOLEANtrueโ€”
anchoroptBOOLEANtrueSearch each section from where the PREVIOUS one's words actually stopped, instead of from where the plan says it begins. Why, measured on a real take: the plan ran +1.5 s behind the performance at the first verse and +17.7 s by the second, growing steadily. A plan is not wrong at random, it is wrong PROGRESSIVELY โ€” so the last section's error is the next section's best correction, and a window lands right on the first try instead of needing the emergency widening. The half that matters more is the floor. A section cannot begin before the previous one stopped singing, so a section the model never sang can no longer be laid confidently on top of the one that IS being sung there. It gets searched where it belongs, finds nothing, and says so โ€” a wrong answer becoming a reported non-answer. A section whose own confidence comes back low teaches the next one nothing, so one bad placement cannot walk the rest of the song off the music. Turn this off to read every window straight off the plan again.

Outputs (5)

NameTypeDescription
timingKINBURG_ECHO_TIMINGโ€”
planSTRINGโ€”
ass_pathSTRINGโ€”
srt_pathSTRINGโ€”
reportSTRINGโ€”