ComfyUI-AceMusic

English | 日本語 | 简体中文 | 繁體中文 | 한국어 | Tiếng Việt

Multilingual AI music generation nodes for ComfyUI powered by ACE-Step. Generate full songs with lyrics in 19 languages including English, Chinese, Japanese, Korean, and more.

---

If you find this project helpful, please give it a Star!

---

Highlights

• First Full-Featured ACE-Step Integration for ComfyUI - Complete implementation of all ACE-Step capabilities as ComfyUI nodes (15 nodes total)
• Modular Architecture - Separated Settings/Lyrics/Caption nodes eliminate widget ordering issues and improve workflow readability
• Cross-Platform Compatibility - Works on Windows with Python 3.13+ by using soundfile/scipy instead of problematic torchaudio backends
• HeartMuLa Interoperability - Seamlessly chain with HeartMuLa nodes for hybrid AI music workflows
• Production-Ready - Robust input validation with automatic fallbacks prevents runtime errors

Features

• Multilingual Lyrics - Generate music with vocals in 19 languages (English, Chinese, Japanese, Korean, Spanish, etc.)
• Song Structure Control - Use section markers like [Verse], [Chorus], [Bridge] to define song structure
• Style Tags - Control genre, vocal type, mood, tempo, and instruments
• 4-Minute Songs - Generate up to 240 seconds of continuous audio
• Audio Editing - Cover, Repaint, Extend, Edit, and Retake capabilities
• LoRA Support - Load fine-tuned adapters for specialized styles
• HeartMuLa Compatible - Works seamlessly with HeartMuLa nodes

Nodes

| Node | Description | |------|-------------| | Model Loader | Downloads and caches ACE-Step models | | Settings | Configure generation parameters (duration, language, BPM, etc.) | | Generator | Generate music from caption and lyrics (Text2Music) | | Lyrics Input | Dedicated node for entering lyrics with section markers | | Caption Input | Dedicated node for style/genre description | | Cover | Transform existing audio into different styles (Audio2Audio) | | Repaint | Regenerate specific sections of audio | | Retake | Create variations of existing audio | | Extend | Add new content to beginning or end of audio | | Edit | Change tags/lyrics while preserving melody (FlowEdit) | | Conditioning | Combine parameters into conditioning object | | Generator (from Cond) | Generate from conditioning object | | Load LoRA | Load fine-tuned LoRA adapters | | Understand | Measure audio duration (caption/BPM/key are placeholders*) | | Create Sample | Generate parameters via keyword heuristics* |

* Full AI-powered audio analysis and parameter generation require a future ACE-Step version. Current implementation provides accurate duration measurement and keyword-based inference as placeholders.

Installation

ComfyUI Manager (Recommended)

Search for "ComfyUI-AceMusic" and install.

Manual

cd ComfyUI/custom_nodes
git clone https://github.com/hiroki-abe-58/ComfyUI-AceMusic.git
cd ComfyUI-AceMusic
pip install -r requirements.txt

Install ACE-Step

pip install git+https://github.com/ace-step/ACE-Step.git

Models are automatically downloaded from Hugging Face on first use.

Quick Start

1. Add AceMusic Model Loader node and select device (cuda)
2. Add AceMusic Settings node to configure parameters
3. Add AceMusic Lyrics Input node and enter lyrics:

   [Verse]
   Walking down the empty street
   Thinking about you and me
   
   [Chorus]
   We belong together
   Now and forever
   

4. Add AceMusic Caption Input with style tags: pop, female vocal, energetic
5. Connect all to AceMusic Generator -> Preview Audio

Load the example workflow from workflow/AceMusic_Lyrics_v3.json

Section Markers

ACE-Step supports these section markers for song structure:

| Marker | Usage | |--------|-------| | [Intro] | Opening instrumental or vocal intro | | [Verse] | Main verses | | [Pre-Chorus] | Build-up before chorus | | [Chorus] | Main hook/chorus | | [Bridge] | Contrasting section | | [Outro] | Ending section | | [Instrumental] | Non-vocal sections |

Style Tags

Combine tags in the caption to control output style:

• Genre: pop, rock, electronic, jazz, classical, hip-hop, r&b, country, folk, metal, indie, j-pop, k-pop
• Vocal: female vocal, male vocal, duet, choir, instrumental
• Mood: energetic, melancholic, uplifting, calm, aggressive, romantic, dreamy, dark
• Tempo: slow, medium, fast
• Instruments: piano, guitar, drums, synth, strings, bass, violin, saxophone

Example: j-pop, female vocal, energetic, bright synthesizer, catchy melody

Models & Hardware

Models download automatically from Hugging Face to ~/.cache/ace-step/checkpoints/

Performance

| Device | RTF (27 steps) | Time for 1 min audio | |--------|----------------|---------------------| | RTX 5090 | ~50x | ~1.2s | | RTX 4090 | 34.48x | 1.74s | | A100 | 27.27x | 2.20s | | RTX 3090 | 12.76x | 4.70s | | M2 Max | 2.27x | 26.43s |

VRAM Requirements

| Mode | VRAM | Notes | |------|------|-------| | Normal | 8GB+ | Full speed | | CPU Offload | ~4GB | Slower but works on limited VRAM |

Parameters

Settings Node

| Parameter | Default | Range | Description | |-----------|---------|-------|-------------| | duration | 30 | 5-240 | Audio length in seconds | | vocal_language | ja | 19 languages | Language for vocals | | bpm | 120 | 0-300 | Beats per minute (0 = auto) | | timesignature | 4/4 | Various | Time signature | | keyscale | (auto) | 24 keys | Musical key | | instrumental | false | bool | Generate without vocals | | inference_steps | 27 | 1-100 | Quality vs speed | | guidance_scale | 15.0 | 1-30 | Prompt adherence | | seed | -1 | int | Random seed (-1 = random) |

Supported Languages

ACE-Step supports 19 languages. Top performers:

| Language | Code | Quality | |----------|------|---------| | English | en | Excellent | | Chinese | zh | Excellent | | Japanese | ja | Excellent | | Korean | ko | Very Good | | Spanish | es | Very Good | | German | de | Good | | French | fr | Good | | Portuguese | pt | Good | | Italian | it | Good | | Russian | ru | Good |

Integration with HeartMuLa

The AUDIO type is compatible with HeartMuLa outputs:

• Use HeartMuLa-generated audio as input to AceMusic Cover
• Use HeartMuLa-generated audio as input to AceMusic Repaint
• Chain HeartMuLa and AceMusic nodes together for advanced workflows

Troubleshooting

ACE-Step installation fails (dependency version errors)

If you see errors like No matching distribution found for torchaudio==2.10.0+cu128 or matplotlib==3.10.1, the ACE-Step repository has strict version requirements that may not be available for your Python version or platform.

Solution: Clone and modify ACE-Step locally

# Clone ACE-Step repository
git clone https://github.com/ace-step/ACE-Step.git
cd ACE-Step

# Edit requirements.txt to relax version constraints
# Change exact versions (==) to minimum versions (>=)
# Example: matplotlib==3.10.1 -> matplotlib>=3.8.0

# Install with relaxed requirements
pip install -e .

Alternative: Install dependencies manually first

pip install torch torchaudio --index-url https://download.pytorch.org/whl/cu121
pip install transformers diffusers accelerate soundfile librosa
pip install git+https://github.com/ace-step/ACE-Step.git --no-deps

Models not loading / Download fails

• Check your internet connection
• Verify Hugging Face access
• Try manually downloading from https://huggingface.co/ACE-Step

Out of VRAM

• Enable cpu_offload in Model Loader
• Reduce duration
• Close other GPU applications

Slow generation

• Enable torch_compile (requires triton)
• Use lower inference_steps (10-15 for drafts)
• Use overlapped_decode for long audio (>48s)

Audio quality issues

• Increase inference_steps (50-100 for best quality)
• Adjust guidance_scale (try 10-20)
• Provide more detailed captions
• Try different seeds

Windows-specific issues

• For torchaudio errors, ensure soundfile is installed: pip install soundfile
• For torch.compile, install triton: pip install triton-windows

Roadmap / Planned Features

The following ACE-Step features are not yet implemented but planned for future releases:

| Feature | Status | Description | |---------|--------|-------------| | Track Separation (Stems) | Planned | Separate audio into vocal/instrumental tracks | | Multi-Track Generation | Planned | Layer generation like Suno Studio "Add Layer" | | Vocal2BGM | Planned | Auto-accompaniment from vocals | | LRC Generation | Planned | Timestamped lyric alignment |

Contributions and PRs are welcome! See Issues for discussion.

Requirements

• Python >= 3.10
• PyTorch >= 2.0.0
• ComfyUI
• ACE-Step

License

Apache 2.0

Credits

• ACE-Step - Original music generation model by ACE Studio and StepFun
• ComfyUI - Node-based UI framework
• HeartMuLa - Inspiration for node design

Links

• ACE-Step GitHub
• ACE-Step Hugging Face
• ACE Studio