Human4D Image2SMPL

Most of the MotionDiff pack generates motion from text. This node goes the other way: it looks at actual video frames and recovers the 3D bodies of every person in them. Human4D Image2SMPL is the pack's image-to-SMPL workhorse - it runs a person detector, then HMR2.0, and hands you an SMPL_MULTIPLE_SUBJECTS object with per-frame meshes, 3D keypoints, and camera info for every detected human. This is the 4DHuman half of the pack, and it's the pipeline people use to drive pose/OpenPose ControlNet from a real person's movement instead of a generated one.

The output plugs into the multi-subject render nodes - Render Multiple SMPL Mesh for silhouettes and depth, and Render OpenPose from SMPL Multiple for actual OpenPose skeletons - or into the multi-subject exporter for 3D software. There's no stop on the way to "I extracted my friend's dance from a phone video and turned it into a ControlNet condition."

How it works

Two stages per frame. First, a YOLO person detector (the one you picked in Humans4DLoader) finds bounding boxes. Then HMR2.0 - a modern human mesh recovery network - regresses SMPL vertices, a camera translation, and 2D keypoints from each cropped person, all in a single forward pass per batch. The node collects those into the multi-subject structure, storing focal length and frame size so the renderers know the camera.

The inputs that matter

• human4d_model - from Humans4DLoader.
• image - your image or video frames (a batched tensor, so a frame sequence works fine).
• det_confidence_thresh (0.25) - how confident the person detector must be. Lower it to catch smaller/further people (and pick up more false positives); raise it to cut noise.
• det_iou_thresh (0.7) - NMS threshold; usually leave alone.
• det_batch_size (10) and hmr_batch_size (8) - detector and HMR batch sizes. Lower hmr_batch_size if you're VRAM-limited; raise it on a big card for speed.

There's an optional opt_scorehmr_refiner input for a SCORE_HMR_MODEL. Fun fact: it's wired up in the UI but the implementation literally raises NotImplementedError if you connect it. Don't bother - it's a stub.

Installation

Part of Fannovel16/ComfyUI-MotionDiff. ComfyUI Manager → "ComfyUI MotionDiff", or clone into custom_nodes, install requirements, restart. Heavy install: smplx, pyrender, timm, scikit-image, webdataset, opencv and more, and the first run auto-downloads the HMR2.0 checkpoint and YOLO weights (from the ultralytics assets / adetailer repos) - a big download, then fast.

Gotchas

The download surprise is the main one: the HMR2.0 checkpoint (a ~1GB model) plus detector weights all come down on first use. Don't panic when the first queue run sits.

Performance is the other. This is per-frame, per-person network inference - a 196-frame clip with several people takes a while. Lower hmr_batch_size if the GPU runs out of memory, and expect the detector to occasionally double-count or miss a person, which is normal for HMR-grade pipelines. It's research-tool territory: impressive when it works, and it usually does, but not polished.