Slimy_HMR2_VNCCS

Slimy_HMR2_VNCCS

A ComfyUI custom node that converts HMR2 keypoints_3d output into Pose JSON format compatible with VNCCS PoseStudio. Detects people in a photo, extracts pose keypoints, and uses HMR2 to infer 3D depth for each joint — automatically generating pose data ready to use in VNCCS PoseStudio.

[!WARNING] This node requires Slimy_HMR2_keyPoint3D to be used together.(https://github.com/Slimy-Comfy/Slimy_HMR2_keyPoint3D) On the first run of that node, approximately 2.5 GB of model data will be automatically downloaded, which may take a significant amount of time. Also, you will need to download the model data from a specific site. Please read the instructions about its repositry carefully. After that, inference runs in approximately 6 seconds.

---

What It Does

Reads 3D keypoints estimated by HMR2 (a human pose estimation AI) and automatically generates bone rotation values (Euler angles) suited for PoseStudio mannequins.

---

How to Use

1. Connect the HMR2 node output (keypoints_json) to the input of this node.
2. Run the node — the conversion result will appear in the text area on screen.
3. Press the Export VNCCS PoseData button to download a Pose JSON file that can be loaded into PoseStudio.

Output Sockets

| Socket | Description | |---|---| | VNCCS_POSE | Clean Pose JSON for PoseStudio | | debug_json | Full data including debug info (KPf coordinates) |

---

Processing Pipeline

HMR2 keypoints_3d (SMPL coordinate system)
    ↓ fit_kp_to_mannequin()
fittedKP (stick-figure coordinates normalized to mannequin space)
    ↓ convert_fitted_to_pose()
Pose JSON (bone rotation values for PoseStudio + modelRotation)

---

Coordinate Systems

HMR2 and PoseStudio use different coordinate systems. The conversion corrects for this automatically.

| System | X | Y | Z | |---|---|---|---| | SMPL KP (HMR2 output) | Right | Down | Away from camera | | Mannequin (PoseStudio) | Right | Up | Away from camera |

Conversion formula: v_mq = [vx, -vy, -vz]

---

Technical Details

Step 1: fit_kp_to_mannequin()

Normalizes raw HMR2 keypoints into mannequin space.

1. Coordinate conversion — Flip Y and Z axes
2. Uniform scale — Apply smpl_scale = mannequin height / SMPL reference height (1.5m) to all XYZ axes
3. Translation — Shift the whole body so the pelvis aligns with the mannequin's Apose position
4. hip_raw storage — Save raw KP hip positions as right_hip_raw / left_hip_raw (used for pelvis front/back detection)
5. Bone length normalization — Rescale from shoulder/hip origin to match mannequin bone lengths
6. Clavicle interpolation — Since SMPL has no clavicle KP, interpolate from 85% along the pelvis-neck line with an X-axis offset

---

Step 2: convert_fitted_to_pose()

Computes local rotation for each bone from fittedKP.

Bone Hierarchy

pelvis
├── thigh_r → calf_r
├── thigh_l → calf_l
└── spine_03
    ├── neck_01 → head
    ├── clavicle_r → upperarm_r → lowerarm_r
    └── clavicle_l → upperarm_l → lowerarm_l

Pelvis Rotation

Faithfully reflects the 2D layout of KPs. No gravity assumption.

right   = normalize(left_hip_raw - right_hip_raw)
up      = pelvis → neck direction (orthogonalized by removing right component)
forward = cross(right, up)  ← front/back determined by winding order of pelvic triangle

Outputs the delta from the Apose rest reference (PELVIS_REST_ROT) as local rotation. A correction offset of -19 degrees is applied to the X axis.

spine_03 Rotation

World rotation matrix computed directly via compute_spine03_rotation().

right   = normalize(left_shoulder - right_shoulder)
up      = spine_03 (FK position) → mid_shoulder direction (right component removed)
forward = cross(right, up)  ← sign determined by pelvis_forward

Local rotation = pelvis_world_q.inv * spine03_world_q

neck_01 Rotation

target = mid_ear(KPf) - head_forward * HEAD_PIVOT_OFFSET

HEAD_PIVOT_OFFSET (default: 0.65) shifts the target toward the back of the head to prevent the head from jutting forward.

Local rotation = spine03_world_q.inv * neck01_world_q

head Rotation

World rotation matrix computed from face KPs (nose / right_ear / left_ear).

right   = left_ear → right_ear
forward = pivot → nose  (pivot is offset backward from mid_ear by HEAD_PIVOT_OFFSET)
up      = cross(forward, right)

HEAD_X_OFFSET (default: 12.0) is added to X rotation as a downward correction. Local rotation = neck01_world_q.inv * head_world_q

clavicle / upperarm / lowerarm / thigh / calf (Common FK)

currentDir = bone direction in Apose (rotated by current world_q)
targetDir  = FK origin position → fittedKP end KP direction
deltaQ     = shortest rotation from currentDir → targetDir
newWorldQ  = deltaQ * currentBoneWorldQ
localQ     = parentWorldQ.inv * newWorldQ

modelRotation

Converts the human body's forward direction from global_orient (SMPL global orientation matrix) into mannequin space, and outputs the Y-axis angle difference from camera front (-Z) as modelRotation[1].

---

Calibration Constants

The following constants at the top of the code can be adjusted for calibration.

| Constant | Default | Description | |---|---|---| | HEAD_PIVOT_OFFSET | 0.65 | Amount to shift neck_01 target toward the back of the head (in mannequin scale). Larger values tilt the neck further back. | | HEAD_X_OFFSET | 12.0 | X rotation correction for the head bone (degrees). Higher values tilt the head further down. |

---

Output JSON Format

{
  "type": "single_pose",
  "version": "1.0",
  "bones": {
    "pelvis":     [rx, ry, rz],
    "thigh_r":    [rx, ry, rz],
    "thigh_l":    [rx, ry, rz],
    "calf_r":     [rx, ry, rz],
    "calf_l":     [rx, ry, rz],
    "spine_03":   [rx, ry, rz],
    "neck_01":    [rx, ry, rz],
    "head":       [rx, ry, rz],
    "clavicle_r": [rx, ry, rz],
    "clavicle_l": [rx, ry, rz],
    "upperarm_r": [rx, ry, rz],
    "upperarm_l": [rx, ry, rz],
    "lowerarm_r": [rx, ry, rz],
    "lowerarm_l": [rx, ry, rz]
  },
  "modelRotation": [0, Y, 0]
}

bones values are Euler angles (degrees) in Three.js XYZ order.

---

Mannequin Definition (Apose Measured Values)

MANNEQUIN_APOSE = {
    'pelvis':         [ 0.000,  7.362,  0.397],
    'right_hip':      [-0.817,  7.088,  0.492],
    'left_hip':       [ 0.817,  7.088,  0.492],
    'right_knee':     [-1.116,  3.834,  0.673],
    'left_knee':      [ 1.116,  3.834,  0.673],
    'right_ankle':    [-1.328,  0.560,  0.254],
    'left_ankle':     [ 1.328,  0.560,  0.254],
    'spine_01':       [ 0.000,  7.639,  0.082],
    'spine_02':       [ 0.000,  8.143,  0.208],
    'spine_03':       [ 0.000,  8.758,  0.322],
    'right_clavicle': [-0.211, 10.697,  0.422],
    'left_clavicle':  [ 0.211, 10.697,  0.422],
    'right_shoulder': [-1.242, 10.503,  0.404],
    'left_shoulder':  [ 1.242, 10.503,  0.404],
    'right_elbow':    [-2.549,  9.042,  0.351],
    'left_elbow':     [ 2.549,  9.042,  0.351],
    'right_wrist':    [-3.517,  8.094,  1.653],
    'left_wrist':     [ 3.517,  8.094,  1.653],
    'neck_01':        [ 0.000, 11.263,  0.282],
    'head':           [ 0.000, 12.144,  0.572],
}

If you change the mannequin model, update these values with new measured data. Outdated values will cause bone rotation errors.

---

Known Limitations

• thigh / calf — FK only; accuracy drops for large knee bends where IK would be needed
• clavicle — No KP available, so interpolated from Apose position; errors increase for poses with large shoulder elevation
• pelvis X/Z rotation — Computed from KP layout; errors increase for strong forward lean or lateral tilt
• Twist DOF — Twist (torsion) for each bone is unresolved; downstream IK correction is assumed
• Face orientation — HMR2 has low Z-axis accuracy, so the forward direction (Z) of the face is prone to error