ComfyUI-MuseTalk_FSH

the comfyui custom node of MuseTalk to make audio driven videos!

<div> <figure> <img alt='webpage' src="web.png?raw=true" width="600px"/> <figure> </div>

How to use

make sure ffmpeg is worked in your commandline for Linux

apt update
apt install ffmpeg

for Windows,you can install ffmpeg by WingetUI automatically

then!

git clone https://github.com/AIFSH/ComfyUI-MuseTalk_FSH.git
cd ComfyUI-MuseTalk_FSH
pip install -r requirements.txt

mmlab packages

pip install --no-cache-dir -U openmim 
mim install mmengine 
mim install "mmcv>=2.0.1" 
mim install "mmdet>=3.1.0" 
mim install "mmpose>=1.1.0" 

Download weights

You can download weights manually as follows:

1. Download our trained weights.

2. Download the weights of other components:

   • sd-vae-ft-mse
   • whisper
   • dwpose
   • face-parse-bisent
   • resnet18

或者下载MuseTalk.zip， 解压后把子文件夹放入ComfyUI-MuseTalk_FSH/models/目录

Finally, these weights should be organized in models as follows:

ComfyUI-MuseTalk_FSH/models/
├── musetalk
│   └── musetalk.json
│   └── pytorch_model.bin
├── dwpose
│   └── dw-ll_ucoco_384.pth
├── face-parse-bisent
│   ├── 79999_iter.pth
│   └── resnet18-5c106cde.pth
├── sd-vae-ft-mse
│   ├── config.json
│   └── diffusion_pytorch_model.bin
└── whisper
    └── tiny.pt

Tutorial

• Demo on 3060 12GB
• Demo on 4090 24GB

WeChat Group && Donate

<div> <figure> <img alt='Wechat' src="wechat.jpg?raw=true" width="300px"/> <img alt='donate' src="donate.jpg?raw=true" width="300px"/> <figure> </div>

Thanks

• MuseTalk