Dream Talk

This is the whole pack, in a single node. D_DreamTalk is an unofficial ComfyUI port of Alibaba's DreamTalk - the diffusion-based "expressive talking head" from late 2023. You feed it one portrait and one audio clip, pick a mood, and it returns frames of that face lip-syncing to your audio with the chosen expression baked in. No API, no key, no external service: the whole pipeline runs on your GPU.

A word on expectations before you get excited. DreamTalk is a 2023 research model, and it shows: output is a fixed 256px, faces don't blink, and it wants a clean frontal portrait. In the current talking-head arms race it's been overtaken by LivePortrait (but that one can't be audio-driven), EchoMimic, and Hallo. Where DreamTalk still wins is the style control - you can literally pick "angry level 3" or "happy level 3" and get a visibly different performance. For that, nothing else in ComfyUI does it quite like this.

How it works

Internally the node is doing three jobs. First it resamples your audio to 16kHz mono with ffmpeg, then runs it through wav2vec2-large-xlsr-53-english to pull speech features. Those features plus your chosen style clip feed a small diffusion network (denoising_network.pth, ~48MB) that predicts 3D face motion - expression coefficients, sampled with classifier-free guidance. Finally a ~335MB renderer (renderer.pt) paints the face motion onto your cropped image at 25fps. The "talking head" you see is a rendered re-animation, not a warp of the original pixels.

The inputs that matter

The node is a black box with six knobs, and only a few you'll actually touch:

• audio - a dropdown, not a file picker. It lists every .wav, .mp3, and .flac sitting in ComfyUI/input. Drop your file there, refresh, and it appears.
• image - your portrait. It must contain a face dlib can find (see below).
• style_clip - the interesting one. These 24 .mat files are 3DMM motion templates, named like M030_front_happy_level3_001.mat (M/W are training-subject identities, and the emotions run angry through surprised). Default is neutral. This is what sets the performance's mood.
• cfg_scale (default 1.0) - guidance strength for the diffusion motion network. Push it up to lean harder into the style clip; push it down for more natural, relaxed motion. 1.0 is a fine starting point.
• max_gen_len (default 1000) - a cap on how much audio gets processed. If long clips cut off early, raise it; if it's crawling, lower it.
• img_crop (default on) - crops to just the face before rendering. Turn it off to keep the whole image "talking" (that's what the pack's advanced workflow does).

Outputs

Three: images (the frame tensor - wire it into VHS Video Combine or SaveWebm to get an actual file), count (number of frames), and frame_rate (always 25).

Installing it

Easiest route is ComfyUI Manager - search "dreamtalk." Otherwise:

cd ComfyUI/custom_nodes
git clone https://github.com/hay86/ComfyUI_Dreamtalk
cd ComfyUI_Dreamtalk
pip install -r requirements.txt
sudo apt install ffmpeg   # or however you install ffmpeg on your OS

The requirements are heavy and worth knowing about before you start: dlib (a C++ build that's a genuine pain on Windows without a prebuilt wheel), transformers, torchaudio, av, and opencv-python. The README's story is that checkpoints require emailing the original author for academic-only access - the downloads were pulled "in light of the social impact." Reality is friendlier: the node auto-downloads both checkpoints (about 383MB) from the cncbec/dreamtalk Hugging Face mirror into ComfyUI/models/dreamtalk/ on first run if it can't find them. It looks in the pack's checkpoints/ folder first, so you can also place them there manually.

Where people get burned

• "No face detected in the input image" - dlib failed. Use a clear, well-lit frontal face; profile shots and heavy angles won't work.
• Audio not in the dropdown - it's not in ComfyUI/input, or you added it after ComfyUI started. Restart or reload the node list.
• CUDA error - this is GPU-only; the node refuses to run on CPU.
• ffmpeg errors - the binary must be installed and on PATH even though the pack also pulls the ffmpeg-python library. The README and the code don't fully agree here; trust the code.
• First run is slow - it downloads the wav2vec model (~1.2GB) and the checkpoints on top of loading two neural networks. Give it a minute.

Set the resolution expectations and it's a genuinely fun node - an entire talking-head studio collapsed into one box, with mood control nobody else really matches.