VideoPoseEstimator

Feed VideoPoseEstimator a video clip and it hands you everything you need to rebuild the world that video was shot in: per-frame camera poses, depth maps, the horizontal FOV, and a per-pixel confidence map. Historically that pipeline meant structure-from-motion - COLMAP runs, feature matching, hours of tuning, and it still choked on textureless scenes. This node runs VGGT (facebook/VGGT-1B, roughly 5 GB, downloaded via huggingface_hub on first use) and does the whole job in one pass. It's the front door of the pack's video_to_4d_world workflow, and honestly the single most impressive node in the pack.

VGGT is a single transformer that takes image(s) and directly predicts camera extrinsics, intrinsics, and depth - the "Visual Geometry Grounded Transformer" Meta released in March 2025. It caused a genuine moment on r/StableDiffusion: camera reconstruction from plain monocular video with no calibration and no multi-view search. The demo ran on an H100, which spooked people, but it runs fine on consumer cards if you keep the resolution and frame count sane - that's what this node's defaults are tuned for.

The inputs:

• frames - [T,H,W,3] video frames.
• max_frames - default 64. If your clip is longer, it stride-subsamples down to this many frames for VGGT, then SE(3)-interpolates the poses back to full length (depth and confidence get nearest-frame fill). On a 12 GB card, dropping this to 24–32 is your main OOM lever.
• resolution - default 518, the max image dimension fed to VGGT (rounded to a multiple of 14). Lower it to ~384 when VRAM is tight; it's also your speed knob.
• device - auto/cpu/cuda.

Outputs: trajectory ([T,4,4] world-to-camera poses, OpenCV convention +X right, +Y down, +Z forward), depths ([T,H,W] at the input resolution), horizontal_fov (a single FLOAT, the mean across frames), and confidence ([T,H,W]). Those four wires feed the rest of the pipeline - VideoToFusedSplats wants the trajectory, FOV, and depths; MotionMaskFromDepth and TracksToTrajectories want poses and depth.

Here's the trap worth knowing about, and it's a version thing. The README's "Video → 4D World" section tells you the depths are Z-depth and to insert ZDepthToRayDepthNode before anything that expects ray depth. But the current source (v1.1.0) already converts VGGT's Z-depth to radial ray depth internally - check the code around the depth head output. So if you blindly follow the README, you'll apply the ray correction twice and the geometry will bow the other way, most visibly at wide FOVs. Test on your version: if the depths output is already ray-corrected (it should be, on current code), skip the extra node. The Z-depth caveat genuinely applies to other depth sources - just not to this node anymore.

Other real-world issues: VGGT's install is the classic torch-CUDA-mismatch failure (people hit this everywhere; install.py installs it into your ComfyUI Python, which sidesteps most of it). OOM raises a clear error telling you to lower max_frames or resolution. And depth quality is the ceiling on everything downstream - if your scene has motion blur or reflective surfaces, pose and depth confidence both drop, and the README's mention of DepthEdgeFilter for flying pixels is worth heeding.

Install the pack via ComfyUI Manager (search camera-comfyUI, Install, restart; Manager runs requirements.txt + install.py automatically), or by hand:

git clone https://github.com/Alexankharin/camera-comfyUI.git custom_nodes/camera-comfyUI
cd custom_nodes/camera-comfyUI && python install.py

Expect the 5 GB VGGT download on first run and let it finish. One final honest note: the pose-estimation field moves fast - Depth Anything 3 (late 2025) claims to beat VGGT on multi-view accuracy - but this pack is wired around VGGT, and it's very much good enough to make your video walkable. That's the goal, and this node is the unlock.