LeapTalk Generate (portrait + speech)

The node that finally does the work

Audio-driven avatars are the awkward corner of the local video stack: you have speech - Kokoro, Chatterbox, a recording - and you want a still portrait to move its face to it. The community default has been InfiniteTalk, with LongCat Avatar as the quality-but-slower option; both live in Wan's ecosystem and inherit its model sizes and sampling times.

LeapTalk Generate is a different bet on the same job, and it's the one node in this pack that makes pixels: one portrait plus one speech clip in, a 512x512, 25 fps video of that face saying those words out, with your audio muxed in. No API, no key, no cloud. It's an unofficial wrapper around the official LeapTalk recipe - SoulX-FlashHead 1.3B plus wav2vec2-base-960h for audio - and it follows that recipe closely. Fair warning: nobody's talking about it yet - zero reddit threads against InfiniteTalk's 174 - so the documentation is your only company, and it happens to be good.

How it works, in the order it happens

The node doesn't run the model in ComfyUI's process. It writes the portrait as a PNG, your speech unchanged as a float WAV, and a schema-checked job file into a fresh job folder, then runs runtime/leaptalk_job.py with the Python of your registered runtime - which imports a commit-pinned, unmodified LeapTalk checkout and follows the official inference.py in stream mode.

Inside a job: the portrait is center-cropped to 512x512 and encoded once as the static anchor, and the speech is resampled to 16 kHz mono for wav2vec2. Then it's chunk by chunk - an 8 s audio window, one ViBT solver step from the reference-anchored source with the clamped two-latent history, decode, color correction, history updated by a decoder round-trip, 28 new frames each time. Frames stream into ffmpeg as H.264, get trimmed to ceil(audio seconds x 25) and muxed with your audio as AAC, and the file is decoded back and checked before the node hands it over.

Two things. Generation is streaming and constant-memory: a 96.5 s clip took 71 s on the tested RTX 5090 and GPU memory didn't grow with length. And on the default one-shot backend about 23 s of every job is imports and weight loading - which is what LeapTalk Runtime's persistent backend fixes.

The inputs you actually touch

• runtime - a LEAPTALK_RUNTIME link from LeapTalk Runtime. Runtimes are registered outside the workflow; a workflow can only pick an id.
• image - exactly one portrait. A batch of two or more fails validation. Frontal head-and-shoulders photos behave best.
• audio - one clip, mono or stereo, any sample rate; it's muxed into the output untouched.
• decoder - lite_tae (the official default, a 22 MB TAE decoder) or wan_vae (the Wan2.1 VAE: about 3x slower per chunk; the official script compiles it, this node doesn't).
• audio_guidance - leave it at 1.0. Above 1 adds an unconditional call per chunk, so 2x the compute, and the author's own test at 2.0 gave over-sharpened, discoloured lips.
• preview - on by default; shows the last frame of each finished chunk.

Outputs

video is a VIDEO, so wire it into Save Video like anything else. report is a JSON string, worth wiring into a Show Text node once: wall_seconds, chunks, solver_steps, forwards, attention_backend, timings, memory, which weights were applied, frames_sha256, effective backend. When something looks off it usually answers the question.

Installing it

Manager can find it (search "LeapTalk"), or:

cd ComfyUI/custom_nodes
git clone https://github.com/hiroki-abe-58/ComfyUI-LeapTalk

That's the whole ComfyUI side - the package declares no Python dependencies, on purpose. The heavy half is the runtime: a separate Python 3.12 venv with torch 2.7.1+cu128, runtime/requirements-runtime.txt, LeapTalk at commit 5d8fef8, and about 7 GB of weights (SoulX-FlashHead Model_Pro, the LoRA, the audio projection, wav2vec2). Register that venv in ComfyUI/user/leaptalk.runtimes.json, restart, run the Doctor. It's all in docs/SETUP.md.

Where people get burned

• "LeapTalk takes exactly one reference image" - your IMAGE is a batch. Pick one upstream.
• "CUDA is not available" - the runtime venv got a CPU-only torch. Reinstall from the cu128 index.
• A commit-charge message instead of a job - the memory guard refused to load the model. Close something else or unload other models; loosening the guard isn't the recommended fix.
• Memory pressure mid-job on Windows - CUDA allocations count against the commit limit, so "free RAM looks fine" isn't the whole story (docs/BENCHMARKS.md).

Cancelling in ComfyUI works properly - the runtime process gets ended. After any job error the worker is discarded rather than reused, so one failure costs you the next job's startup time too. Use portraits and voices you have consent for, and label the output as AI-generated.