HSyn9DHPE - 9D Pose Estimation

This node does one genuinely cool thing: you feed it a video of a person, and it extracts a full 3D pose with per-joint rotations, then renders that skeleton (or a body mesh) through Houdini. Not the flat 2D OpenPose stick figures you're used to from ControlNet preprocessors - actual 3D joints plus orientation, the kind of data that can drive a rig or be re-rendered from a completely new camera angle. That re-framing bit is the whole pitch: film someone once, then "capture" the same motion from anywhere.

Now the honest part. This is a research project's ComfyUI wrapper, and it shows: essentially no community footprint, zero search impressions, and an install that needs Houdini (paid), a separate conda environment, and the MMPose stack. Not a five-minute job. But if you're doing mocap-style work and Houdini is already in your life, it's the only ComfyUI node I know of that hands you 3D pose plus rotation data out of a plain video.

What "9D" means and how it works

The pack is a front-end for HSyn9DHPE (Houdini Synthetic data for 9D Human Pose Estimation). The 9D is just marketing-speak for 3D joint positions + a 6D rotation representation per joint. Under the hood it's a two-stage pipeline, both stages running in the project's own conda env, not in ComfyUI's Python:

1. 2D detection - an MMPose MMPoseInferencer runs COCO-WholeBody keypoint detection per frame (RTMPose-l, auto-downloaded on first use).
2. 3D lifting - a graph-oriented transformer (attention + graph convolution over the skeleton, trained on Houdini-generated synthetic data from AMASS mocap) lifts the 2D keypoints to 24 SMPL joints in 3D space with per-joint rotations.

The pose data gets written out as per-frame JSON, and the node then drives a bundled Houdini scene (houdini/houdini_render.hiplc) to render keypoints or a body mesh into ComfyUI/output/HSyn9DHPE/.

The inputs that actually matter

• video (VIDEO, required) - the only thing you must wire in. The node reaches into ComfyUI's built-in video loader object for the underlying file path (it pokes at a private attribute), so feed it from ComfyUI's own video-loading node rather than a third-party suite's output. Feed it a clip of a single visible person; this is per-frame full-body estimation, not multi-person tracking.
• use_hython (default true) - renders headlessly via Hython, Houdini's Python interpreter. Flip it off and open_houdini takes over: Houdini launches in GUI mode so you can re-frame the render camera, then hit the render button yourself. That's the camera-rewrite feature.
• smooth_detections and smooth_3d (both default off) - turn these on when the source footage is jittery. Detection smoothing filters outlier 2D detections; 3D smoothing applies Savitzky-Golay to positions and SLERP to rotations. Leave them off for clean, well-lit footage - they cost time and can flatten real motion.

The outputs

Four outputs, all pointing into ComfyUI/output/HSyn9DHPE/: keypoints_render_dir and body_render_dir (folders of rendered frames - wire them into a video loader to rebuild the clip), start_image_path (a JPEG of the first frame, handy as a starting condition for a generation pass), and total_frames (an INT you can use to sync downstream loops).

Installing it

The README's full ceremony: clone the pack into custom_nodes, clone the HSyn9DHPE core repo inside it, create the conda env with conda env create -f environment.yml -p ./env, mim install mmcv==2.1.0, download an MMPose model, edit hardcoded Houdini paths, restart. The core commands:

cd ComfyUI/custom_nodes
git clone https://github.com/edoardocompagnucci/ComfyUI-HSyn9DHPE.git
cd ComfyUI-HSyn9DHPE
git clone https://github.com/edoardocompagnucci/HSyn9DHPE.git
cd HSyn9DHPE && conda env create -f environment.yml -p ./env
mim install mmcv==2.1.0

Note this is a Windows-first setup: the pack hardcodes env/python.exe and Houdini 20.5.445 paths.

Where people get burned

The README's install steps don't match the shipped code, and it will bite you. It tells you to run mim download mmpose --config td-hm_hrnet-w48_8xb32-210e_coco-wholebody-384x288, but that's a different 2D detector than the one inference actually uses (RTMPose-l, downloaded automatically). More importantly, the node looks for checkpoints/model.pth - the GraphFormer 3D-lifter weights - and the HSyn9DHPE project's own README says those weights are not yet publicly released. So a fresh install will almost certainly die with "Checkpoint not found" until the author ships model.pth or you train your own. On top of that, every run wipes the whole output/HSyn9DHPE directory before writing fresh results, and missing or mis-pointed Houdini paths fail with a cryptic subprocess error. If you don't already own Houdini, treat this as a research curiosity and come back when the model weights land.