SAM3D From Video JK🐉

If you've ever wanted to take a video of a person dancing and get a per-frame 3D body mesh out of it - for character animation, motion transfer, or just to stop relying on 2D pose skeletons - this is the node. SAM3D From Video JK runs SAM3D Body over an image batch (your video frames), reconstructs a mesh for each frame, smooths the sequence over time, and writes a binary mesh-sequence file you can feed straight into Adv3DViewer JK for inspection.

How it works

You feed it a SAM3D_MODEL (loaded by the SAM3D Body pack's own loader) and an IMAGE batch - the frames of your video. It detects the person in each frame, reconstructs the body mesh, and aligns the sequence. A few optional knobs tune the result:

• bbox_threshold - the human-detection confidence threshold (default 0.8).
• inference_type - full, body, or hand. Full body is the default; hand gives you hand-only meshes, which matters if your downstream task is hand tracking.
• mask - an optional mask to constrain which region gets processed.
• smoothing_method / smoothing_sigma - temporal smoothing across frames. gaussian with a larger smoothing_sigma gives a smoother surface but blurs fast motion; moving_average is the cheaper alternative. This is the knob that separates "jittery per-frame reconstruction" from "usable motion."

The output is bin_file_path - a string pointing at the binary mesh-sequence file, saved in the Adv3DViewer_JK_tmp directory under your output folder. Wire that path into Adv3DViewer JK to view it.

Inputs and outputs

• model - SAM3D_MODEL, from the SAM3D Body pack's loader.
• image - IMAGE batch, your video frames.
• output_filename - the name of the saved binary file.
• Optional: bbox_threshold, inference_type, mask, smoothing_method, smoothing_sigma.
• bin_file_path - STRING, the path to the saved mesh sequence.

Installing it

This node has a hard external dependency that the pack does not install for you. From the README: you need ComfyUI-SAM3DBody installed (plus whatever model it downloads). Without it, the node prints a warning and won't run properly.

First install ComfyUI-JakeUpgrade:

• ComfyUI Manager → search "JakeUpgrade" → install → restart.
• Or:

cd ComfyUI/custom_nodes
git clone https://github.com/jakechai/ComfyUI-JakeUpgrade
cd ComfyUI-JakeUpgrade
pip install -r requirements.txt

Windows portable: run install.bat or ../../../python_embeded/python.exe -s -m pip install -r requirements.txt.

Then install ComfyUI-SAM3DBody the same way (Manager search, or git clone https://github.com/PozzettiAndrea/ComfyUI-SAM3DBody into custom_nodes), and follow its model-download instructions.

Common issues

• "SAM3D dependency unavailable" warning. The SAM3DBody pack (or its model) isn't loaded. This is the #1 failure, and it's expected - the dependency isn't bundled.
• Missing detections or bad alignment. Lower bbox_threshold if the person isn't being found; if the mesh sequence drifts relative to the video, that's alignment - recent pack versions align the sequence to the video's camera and zero the z-offset, so update the pack if you're on an old build.
• Jittery meshes. Raise smoothing_sigma (gaussian) or switch methods. Watch for over-smoothing: a sigma that's too high will turn a quick gesture into mush.
• It's for human bodies. The model is a body tracker - full/body/hand are the only inference types. It's not general 3D scene reconstruction.

This is a specialist motion-capture tool, sitting at the top of JakeUpgrade's "character motion model" pipeline: video → SAM3D body meshes → view/export. Heady stuff, but the node keeps the machinery hidden - you mostly just point it at frames and wait.