Float_Animator

The talking-portrait task - feed in a photo of a face and an audio clip, get back a video where the mouth moves in sync with the words - usually means chaining half a dozen nodes together. FLOAT cuts that down to one. Float_Animator wraps the FLOAT model from deepbrainai-research so that a Load Image and a Load Audio are basically all you need to produce a lip-synced portrait clip, with a bonus: it can also steer the face's emotion to match or override what the audio says.

It sits in a crowded niche. LivePortrait (expression transfer from a driving video) and SONIC are the usual alternatives, and the community verdict on FLOAT is consistently "it does its job": quick, decent quality, but narrower than the others. Where people actually get burned is the input constraints - more on that below.

How it works

FLOAT (from the paper Generative Motion Latent Flow Matching for Audio-driven Talking Portrait, arXiv 2412.01064) is a flow-matching model. Instead of a diffusion denoise loop, it learns a motion latent space and integrates along a learned flow from noise to a realistic talking-head sequence - the node runs this with an Euler ODE solver at 10 function evaluations by default.

Under the hood the pack does the classic three-encoder dance: a wav2vec2 model encodes the audio, a second wav2vec2 fine-tuned for speech emotion recognition reads the emotion out of the voice, and the reference image is encoded separately. A motion transformer (the FMT, 8 blocks) fuses all three into motion latents, and a generator decodes frames. Your reference image and audio are written to temp files under ComfyUI/temp/, inference runs, the model is shoved back to CPU to free your VRAM, and the temp files are cleaned up. It handles the whole load.

The inputs and outputs that matter

Nine inputs, but you'll touch maybe five:

• ref_image (IMAGE) and audio (AUDIO) - the portrait and the driving sound. Wire a Load Image and Load Audio straight in.
• emotion - pick a target style, or leave none to let the model infer emotion from the voice. The choices are the standard six plus: angry, disgust, fear, happy, neutral, sad, surprise.
• fps - output frame rate (default 25).
• auto_crop - off by default; flip it on to crop and scale the face the way the original FLOAT project recommends.

The three *_cfg_scale sliders are classifier-free guidance for each conditioning signal - audio (default 2.0), reference image (1.0), and emotion (1.0). The audio one is what actually drives lip-sync fidelity, so that's the one to nudge if the mouth lags. seed is your reproducibility knob; model picks which float.pth from ComfyUI/models/Float/.

The outputs are what you'd hope: animated_frames (standard IMAGE tensor), audio (your input passed through unchanged), and fps. Feed all three into a VHS_VideoCombine to render an actual video, or Save Image for the frame sequence.

Installing it

Easiest via ComfyUI Manager - search "ComfyUI Float Animator" and install. Or the manual way:

cd ComfyUI/custom_nodes
git clone https://github.com/KERRY-YUAN/ComfyUI_Float_Animator.git
cd ComfyUI_Float_Animator
pip install -r requirements.txt   # use your ComfyUI Python env

The requirements.txt is a genuine pile: torchdiffeq, librosa, transformers, timm, face_alignment, av, gdown, gitpython, and more. On top of that the README warns you to keep torch/torchaudio compatible with your ComfyUI PyTorch install - this is a textbook case of the dependency-hell pattern the ecosystem is infamous for, so consider a clean venv.

Then the models. The node auto-downloads everything into ComfyUI/models/Float/ on first run: float.pth (from Google Drive, hence gdown), plus the wav2vec2-base-960h and wav2vec-english-speech-emotion-recognition folders from Hugging Face. Give it time. If the in-node download fails, run Model_Download.bat on Windows or python model_download/model_download.py.

Gotchas and troubleshooting

• Front-facing, roughly square face, or nothing. This is the big one, and it's community-confirmed: FLOAT only works well with full-on front-facing faces. There's no head-pose reenactment here, so a three-quarter view gives suboptimal results. Because it crops only the face, users routinely generate the talking close-up and then blend it back into a full-body still with inpainting.
• auto_crop is off by default. The original project recommends cropping for best results; enabling it can introduce black padding around the face, which is why the author made it opt-in.
• Clean audio. Background music degrades both lip-sync and the inferred emotion. Use a vocal-extraction tool first if your clip isn't clean.
• Refresh after the first download. The model dropdown won't show float.pth until you reload the page - the node's own tooltip says so.
• Read the license before you ship anything. The wrapper code is MIT, but the FLOAT model and core inference code are CC BY-NC-ND 4.0 - non-commercial, no derivatives. That's a hard wall for client work.

It's a single-purpose node with real constraints, but for its one job - a frontal portrait that reads a script convincingly - it's fast and surprisingly good. Just keep the face square, the audio clean, and your commercial plans elsewhere.