DiffSynth: YuE2 Inference

Local music generation has mostly meant ACE-Step - a music model you prompt like an image, vocals and all, rough around the edges but genuinely useful. YuE2 is the other open answer, and its pitch is different in a way that matters: it writes a symbolic score first, then sings it. You get lyrics, structure and a plan you can edit, not just a render.

What it is

m-a-p/YuE2-3B plus m-a-p/YuE2-Vae, from the Multimodal Art Projection group. You hand it a style prompt and lyrics; it produces a complete song with vocals and accompaniment. The interesting design choice is the chain-of-thought: it plans melody and chords as an ABC-notation score before synthesising audio, which is why the node exposes a cot selector and an abc field. A reharmonised or hand-written score can be fed straight back in.

Two things to know before you get attached. The weights are CC-BY-NC-4.0 - non-commercial. And this pipeline is not tiny: YuE2's own benchmarks put it well above a casual VRAM budget when you lengthen the song, which is what the decode knobs here are for.

The inputs that matter

• pipe - required, from DiffSynth: YuE2 Loader.
• prompt - the style prompt, not a scene description. Think "English, warm piano pop, expressive female voice, acoustic piano, rounded bass and light drums, lyrical memorable melody, unhurried phrasing, 88 BPM".
• lyrics - multi-line, and structure goes in here. [Verse] and [Chorus] tags shape the song the way tags shape a prompt.
• seed (default 831001), num_inference_steps (30) - the generation pair.
• cot - the planning mode, and the most interesting widget on the node. full plans melody plus chords (the default), melody plans melody only and is what you want for covers, off skips planning entirely and throws everything at the audio model. off has no symbolic plan to inspect or edit.
• abc - feed in your own ABC-notation score. Leave empty and let the model write one.
• cfg_scale - text guidance. Leave it empty and the pipeline picks its own sensible value per mode rather than making you guess.
• vae_core_frames - the audio VAE decodes in tiles; this is the tile size. Lower it when decoding is what's eating your VRAM.

Output is a single audio socket, wired to Save Audio or any audio node.

How it runs

Planning pass → autoregressive generation of the song tokens → tiled VAE decode to a waveform. That middle stage is the slow one and it narrates token counts in the terminal, which is genuinely useful for knowing whether you're looking at two minutes or ten. The decode stage happens chunk-wise with a halo, so vae_core_frames trades VRAM against time without changing the song.

Install

Manager → search DiffSynth-ComfyUI → install → restart. Manual:

pip install git+https://github.com/modelscope/DiffSynth-Studio.git
cd ComfyUI/custom_nodes && git clone https://github.com/modelscope/DiffSynth-ComfyUI.git

Weights are m-a-p/YuE2-3B and m-a-p/YuE2-Vae - a few gigabytes, fetched by the ModelConfig nodes on first queue.

Where people trip

• A style prompt where lyrics should go. The two fields do different jobs; "lo-fi hip hop" in lyrics produces nonsense, and a verse in prompt produces a very confused style.
• Expecting ComfyUI's node to be faster than the reference implementation. It isn't doing anything clever; the AR stage is the AR stage.
• cot=off then wondering where the score went. There isn't one - that's the mode.
• Ignoring the licence. CC-BY-NC. Personal projects fine, client work is a conversation.
• Framework drift. Inputs are introspected from your installed diffsynth at startup; restart ComfyUI after upgrading the library.