RealisDanceNode

RealisDanceNode is the ComfyUI port of RealisDance, the controllable character-animation model from Alibaba's DAMO Academy (thefoxofsky's project). Feed it a reference image of a character plus a dance motion, and it renders that character doing the motion. If that sounds like Wan Animate or SCAIL-2 territory, you're right - RealisDance is the earlier, SD 1.5-era take on the same idea, and it predates the whole Wan motion-transfer wave. What makes it unusual is what it uses for the motion: not just a pose skeleton, but 3D hand and body meshes on top of it.

Here's the honest opener, though: this is a one-day port from September 2024 that's been abandoned since the day after it was created. It works if you can feed it, and the feeding is the hard part.

What it actually is

RealisDance generates a video of your reference character performing the motion captured in three conditioning inputs: a DWPose keypoint file (2D skeleton per frame), a HAMER mesh video (3D hand and body reconstruction), and an SMPL mesh video (3D body surface). All three are driving signals - the model reads the pose and the meshes and re-renders the person from your image in those positions.

There is no prompt input. None. The pipeline call takes reference image, pose, meshes, and a seed - text conditioning isn't in the model. That's fine for what it does, but it's a shock if you're used to every ComfyUI node having a CLIP field. The character's appearance comes entirely from the reference image, injected through DINOv2 features and a reference-UNet with mutual self-attention (the magic-animate trick). The motion comes from the pose/mesh trio, fused by a pose guider, with AnimateDiff-style temporal motion modules on top of an SD 1.5 UNet (Realistic Vision 5.1). Scheduler is DDIM with zero-SNR v-prediction, guidance scale 2, at 768×576 and up to 80 frames sampled every fourth frame of your driving material.

Key practical note: your driving videos should be ~4× longer than the output, since the node strides through every fourth frame - a 320-frame source gives an 80-frame result, about 10 seconds at the default fps 8.

The inputs that matter

• dwpose_path, hamer_path, smpl_path (PATH): the three driving files. dwpose is a pickled .pkl of per-frame OpenPose-style keypoint dicts; hamer and smpl are .mp4 renders of the 3D meshes. All three must have the same number of frames or the node throws a length assertion. This trio is the gate - see troubleshooting.
• ref_image (IMAGE): your character, straight from LoadImage. This is what the output person will look like.
• fps (INT, default 8): playback rate written into the output mp4. Raise it if your source motion was captured at a higher frame rate.
• mixed_precision (fp16/bf16) and seed (default 42): bf16 is the safer pick on Ampere+ cards; seed keeps a result reproducible.

Outputs are two VIDEO paths saved to your ComfyUI output folder: sample_video (the generated dance) and refer_sequnces (misspelled in the source, that's the pack's charm) - a side-by-side grid of your reference frame, drawn pose, hamer, and smpl videos. That second one is your sanity check: if the grid looks misaligned, your conditioning was misaligned. Wire both into PreViewVideo from the same pack to actually see them.

Installing and the first-run download

ComfyUI Manager can find it as "RealisDance-ComfyUI", or:

cd ComfyUI/custom_nodes
git clone https://github.com/AIFSH/RealisDance-ComfyUI

Then restart ComfyUI and be patient. The first time you add the node, its __init__ auto-downloads three Hugging Face repos into pretrained_models/ - about 15 GB total:

• SG161222/Realistic_Vision_V5.1_noVAE (~4 GB, the SD 1.5 base)
• facebook/dinov2-large (~1.2 GB, the reference encoder)
• theFoxofSky/RealisDance (~9 GB, the stage_2_hamer_release.ckpt weights)

The requirements.txt is a training-grade list - deepspeed, wandb, accelerate - pulled in for inference too, and it pins diffusers==0.30.1, which can conflict with other packs expecting a newer diffusers. The code is hardcoded to CUDA (vae.to("cuda")), so no CPU, no Apple Silicon; budget 12 GB+ VRAM to be comfortable.

Where people get burned

The real blocker is the input trio. There is no node in this pack that produces the hamer and smpl driving videos - those come from external HAMER and SMPLer-X pipelines from the original research project. The pack ships demo files (demo/dwpose_1.pkl, demo/hamer_1.mp4, demo/smpl_1.mp4) in its own folder, and the open GitHub issue asking how to generate the driving videos is exactly the question you'll have. Practical path: run the demo trio once to confirm the install, and treat generating your own HAMER/SMPL inputs as an external research-pipeline project, not a ComfyUI task. The pack's been dormant since September 13, 2024 - don't expect that gap to close.