FL YuE2 · Gemini Music Captioner

What it is and why you'd reach for it

Captioning is the least glamorous part of any LoRA run and usually the one that decides whether it works. For images you point JoyCaption at a folder and go. For music you're asking a model to describe genre, instrumentation, vocal character and transcribe the lyrics verbatim, which is a much nastier job - so this node calls Google's Gemini, which listens to the actual audio.

That's the honest framing: this one really does call an API, and it really does cost money. It uploads each recording to Google and bills you for the tokens. If that's a dealbreaker, caption by hand - the sidecar format is just two text files, and this node's only job is to fill them in for you.

How it works

Queue it and it walks the audio folder, uploads each file to Google's file service, waits for processing, and asks for a schema-constrained JSON response rather than prose: style, lyrics, instrumental, uncertainty, complete. Constraining the output is why this behaves like a tool instead of a chatbot - you can't get a paragraph of commentary where a caption should be.

Then it writes the result where the rest of the pipeline expects it: <name>.caption.txt, <name>.lyrics.txt, and a <name>.caption.json marker carrying the audio hash, the uncertainty notes and a reviewed: false flag. That marker is what stops Dataset Maker from quietly training on hallucinated lyrics.

Two mechanisms worth knowing about, because they explain almost every message you'll see:

Long songs get chunked. If Gemini reports it couldn't cover the whole recording, the node splits it into roughly three-minute windows with three seconds of overlap and transcribes each with instructions about which interval to actually write down, then stitches them together. A window that still fails gets halved, down to a 15-second floor, before the node gives up and names the interval. So "shorten the input into reviewed song sections" isn't a broken feature - it means Gemini genuinely couldn't hear that part.

Failures are classified. Transient API errors (429 and the 5xx family) are retried up to three times with backoff. Refusals and schema failures are surfaced immediately rather than retried, because retrying those burns your quota for nothing.

Results are cached per folder and reused unless you ask for regeneration, which is why a second queue of the same graph is nearly instant.

The inputs that matter

audio_directory - same convention as the rest of the pack: relative to ComfyUI/input, or absolute.

api_key - the Google Gemini key. Read the tooltip twice: machine environment keys are not used, and normal ComfyUI widgets are saved into your workflow JSON and history. Clear the field before you share a graph. The pack at least passes the key to its worker over stdin rather than writing it into job files on disk, which is a nice touch.

model - three options: gemini-3.8-flash, gemini-3.1-pro-preview and gemini-2.5-pro. Flash for bulk, pro for the messy cases. Model IDs live on Google's schedule, not this pack's; if a request fails mentioning the selected model, that's the thing to change.

task - both, style or lyrics. If you only need style captions (instrumentals, no vocals to transcribe), style is cheaper and one less thing to hallucinate.

concurrent_requests - default 3, range 1–8. Drop it to 1 when Google starts rate-limiting you. Excerpts within a single song are always processed in order, so raising this doesn't scramble your lyrics.

instructions and replace_existing are the two you'll use for taste: steer how genre gets described, and force a re-run on files you already captioned.

The output is YUE2_CAPTIONS, a handle to the manifest - but the real product is the sidecar text files on disk. This is also an output node, so you get the review panel in the UI. Use it.

Install

Install the pack via ComfyUI Manager (search ComfyUI-FL-YuE2) or by cloning it, then install the training requirements - that's where google-genai lives:

cd ComfyUI/custom_nodes
git clone https://github.com/filliptm/ComfyUI-FL-YuE2.git
cd ComfyUI-FL-YuE2
python -m pip install -r requirements.txt
python -m pip install -r requirements-training.txt   # google-genai>=2.19.0, transformers, soundfile, scipy, demucs

Use ComfyUI's own Python interpreter and restart after installing. The pack reuses ComfyUI's Torch and does not install the upstream YuE2 package. No model downloads happen here; captioning needs no GPU, only network access.

Where people get burned

You still have to review. Dataset Maker refuses any recording whose .caption.json says reviewed: false. Every generated caption starts that way, deliberately. Skim the lyrics, fix the obvious nonsense, mark it reviewed - training on an unreviewed transcript is how you teach a model to sing the wrong words confidently.

Processed vocals get misheard. The pack's own validation notes say Gemini transcription can miss or misinterpret processed vocals - heavy reverb, pitch-shifting and dense mixes are all riskier. That's also the argument for cursor_weight 0 later in Train Config if you know the transcript is shaky.

Quota errors. Rate limits are the one thing concurrency directly controls. Setting concurrent_requests to 1 trades wall-clock time for far fewer 429s.

Changed audio, stale captions. Swap a file after captioning and the node notices the hash mismatch, making you either regenerate or supply reviewed sidecars by hand. The pack is unusually paranoid about data drift all the way down the pipeline, and it saves you from silent, mystifying training runs.