DiffSynth: YuE2 Loader

Music models are loaders with a rhythm section. This one assembles YuE2 - the 3B song model plus its VAE - and hands the inference node a pipe that carries everything: the transformer, the codec, and the tokenizer and generation config the autoregressive stage needs to speak YuE2's dialect.

Inputs

Required:

• torch_dtype - bfloat16 (default), float16, float32.
• device - cuda or cpu.
• model_configs - merged from Merge ModelConfigs. Required.

Optional, and genuinely not optional in practice:

• tokenizer_config - the YuE2 tokenizer. The framework's own default is m-a-p/YuE2-3B with pattern qwen.tiktoken, which hints at what's under the hood: YuE2's AR stack rides a Qwen-family tokenizer, which is why qwen.tiktoken is the file you're looking for. Skip it and the pipeline can't build the [Tags] / [Lyrics] text prefix the model expects.
• generation_config - yue2_generation_config.json from the same repo. This carries the sampling defaults for the token stage, so a missing generation config means the pipeline is guessing at how songs should be generated.
• vram_limit - from a VRAM Limit node.

Output: pipe → DiffSynth: YuE2 Inference.

The file list

Four ModelConfigs, from the pack's own YuE2 example graph:

| Into | model_id | origin_file_pattern | |---|---|---| | model_configs | m-a-p/YuE2-3B | model.safetensors | | model_configs | m-a-p/YuE2-Vae | model.safetensors | | tokenizer_config | m-a-p/YuE2-3B | qwen.tiktoken | | generation_config | m-a-p/YuE2-3B | yue2_generation_config.json |

Two of those are large weights and two are tiny text files, and the two tiny ones are the ones that produce confusing failures when absent. The split exists because DiffSynth treats every external artifact - weights, tokenizer, sampling config - as a first-class ModelConfig you can point wherever you like, including a local path on the path field instead of a download.

Files land in models/<model_id>/ relative to ComfyUI's working directory. DIFFSYNTH_MODEL_BASE_PATH moves that before the first download; setting it afterwards moves nothing and you get a second copy.

Install

Manager → search DiffSynth-ComfyUI → install → restart. Manual:

pip install git+https://github.com/modelscope/DiffSynth-Studio.git
cd ComfyUI/custom_nodes && git clone https://github.com/modelscope/DiffSynth-ComfyUI.git

Single dependency: diffsynth>=2.1.9. Audio adds no extra pip requirements - the VAE comes with the weights.

VRAM reality check

A 3B music model plus a codec, and the autoregressive stage scales its memory with song length rather than running at a fixed cost. On a modest card you'll spend your time in the decode stage, which is why the inference node has a vae_core_frames knob and this Loader has a vram_limit input to sanction. If you want to be told how much card the pipeline may claim, wire a VRAM Limit here; if you want it to prefer CPU for the parked weights, that's the VRAM Config going into each ModelConfig.

Common issues

• Tokenisation errors or a song that ignores the lyrics. tokenizer_config missing or pointing at the wrong file. qwen.tiktoken is the one.
• Odd, unstructured generations. generation_config missing; the token stage is falling back to framework defaults.
• "It works but takes forever." Expected. Long songs are long autoregressive sequences; that's the architecture, not a node setting.
• Upgraded diffsynth and the sockets changed. Loader inputs are generated from the installed framework at ComfyUI startup, so restart after any upgrade.