DiffRhythm Run

Type a style prompt, paste some lyrics, and this node hands you a finished song - singing, instrumentation, the works - as a 44.1 kHz audio clip. No API, no key, no cloud. When the pack shipped in spring 2025 the reaction on r/comfyui was basically "holy shit, it's fast." It's the first thing in the ComfyUI ecosystem that made local, end-to-end music generation feel real rather than a research demo.

What it actually is

DiffRhythmRun is the main node in billwuhao's ComfyUI_DiffRhythm pack, a ComfyUI wrapper around ASLP-lab's DiffRhythm model. DiffRhythm is a 1B-parameter diffusion transformer that generates music directly in the mel-spectrogram latent space - no midi, no drum loops, no assembling stems. You describe a vibe and it invents a coherent song, and it does it fast: the upstream project's whole gimmick is that a full-length track generates in about a minute on a decent GPU.

How it works

Under the hood the pipeline chains four pieces: a MuQ-MuLan model turns your style prompt (or a reference audio clip) into a style embedding; an XLM-RoBERTa tokenizer plus a custom Chinese/English grapheme-to-phoneme layer converts lyrics into phoneme tokens; a conditional flow-matching DiT generates the song's latent representation over however many frames the model supports; and a VAE decodes that back to audio. The output is a standard ComfyUI AUDIO tensor at 44100 Hz, so it plugs straight into Save Audio.

Two model sizes exist in the model dropdown. cfm_model_v1_2.pt (the default) and cfm_model.pt generate roughly 95 seconds of song; cfm_full_model.pt jumps to about four-plus minutes. Everything in DiffRhythm is decided by frame count, so pick the model that matches the song length you're making.

The inputs that matter

Most of the defaults are fine, honestly. The ones you'll touch:

• style_prompt - your whole creative brief in one multiline box. "Indie folk ballad, coming-of-age themes, acoustic guitar picking with harmonica interludes" is the built-in example and it's a good template: genre, mood, instruments.
• lyrics_or_edit_lyrics - optional, but this is what separates "song" from "instrumental." The format is LRC-style: each line prefixed with a [mm:ss.xx] timestamp so the model knows when that line starts. Untimestamped lines get silently dropped, and the short models trim the last line, so keep timestamps ascending and inside the song length. Wire the MultiLine Lyrics node in for a nicer editor.
• style_audio_or_edit_song - drop an audio clip in here and the model matches its style instead of using the text prompt (the prompt is then ignored).
• edit + edit_segments - the remix feature. Set edit to true, feed the original song into style_audio_or_edit_song, and regenerate only the seconds listed in edit_segments (default [-1, 20], [60, -1], where -1 means "from the start" or "to the end"). Handy for rewriting one weak verse without re-rolling the whole track.
• seed, steps, cfg, quality_or_speed - sampler controls you'll recognize from image gen. quality mode batches five candidates and picks the best, slower but visibly better on long songs. Note that seed 0 means "random" - set an actual number if you want reproducible takes.

The single output is audio, and that's it. There's no is_output_node here, so remember to add a Save Audio node downstream or your song evaporates into the void.

Installing it

cd ComfyUI/custom_nodes
git clone https://github.com/billwuhao/ComfyUI_DiffRhythm.git
cd ComfyUI_DiffRhythm
pip install -r requirements.txt

ComfyUI Manager also has it - search "DiffRhythm" and install the ComfyUI_DiffRhythm_MW pack. The requirements are heavy (transformers, torchdiffeq, librosa, phonemizer, pyopenjtalk, bitsandbytes, onnxruntime), so the first run can take a while and occasionally collides with other nodes' pinned versions.

The real setup cost is the models. They're not auto-downloaded. You grab each piece manually from the HuggingFace repos linked in the README and drop them into ComfyUI/models/TTS/DiffRhythm/ in the exact folder layout shown: the CFM weights, vae_model.pt, the MuQ-MuLan-large and MuQ-large-msd-iter encoder folders, xlm-roberta-base, and eval-model. Miss one and you get a load error instead of a song. Windows also needs espeak-ng installed with the PHONEMIZER_ESPEAK_LIBRARY environment variable pointed at libespeak-ng.dll; Linux just runs apt-get install espeak-ng. Skip this and lyrics crash on tokenization.

Gotchas

• unload_model defaults to true, so the model is freed from VRAM after every run. Generating several songs back to back reloads it each time - flip it off if you're batching, at the cost of VRAM staying hot.
• It's a 1B DiT plus the encoders loaded simultaneously. It runs on a mid-range GPU fine at 95 seconds, but the full-length model will want more VRAM.
• Import failures are the #1 reported issue, and the recurring fix in the community is simply running a current ComfyUI - old versions trip over this pack's dependencies.

DiffRhythm won't replace your favorite artist. But for a game jam track, a podcast intro, or testing whether that hook in your head actually works, having it one node away from Stable Diffusion is genuinely magical.