3d photo inpainting

The name is a lie in the best way: "3d photo inpainting" isn't a Photoshop tool, it's a one-node pack that takes a single flat photo and renders a real camera move - dolly zoom, zoom-in, circle orbit, swing - as an MP4, entirely on your machine. No API, no key, no cloud. If you've seen those "living photo" parallax clips and wondered how to make them locally, this is a direct answer.

It's a wrapper around the research project of the same name (vt-vl-lab, CVPR 2021), published by AIFSH, the same outfit that wrapped Hallo and OmniGen. Expect that vibe: a faithful but thinly-finished wrapper with rough edges. It's 2021 tech - not state of the art, but it still does one thing most newer tools don't.

How it works

The "inpainting" is the whole trick. A plain depth-warp parallax has a hard limit: your depth map only knows about the original camera view, so as the virtual camera moves it exposes background regions with no pixel data and the image tears. This pipeline solves that properly:

1. Depth estimation with MiDaS (the node forces the heavier BoostingMonocularDepth path off, so just MiDaS).
2. Three trained inpainting networks (edge, depth, color) hallucinate the geometry and the color of the disoccluded regions, building a layered depth image.
3. The result becomes a 3D mesh (it writes a .ply), which Vispy renders from your chosen camera trajectory into the video.

That's why the camera moves can be bigger than a nervous 2% shift. It's also why it's slow and why it wants a CUDA GPU - the whole thing runs as a blocking Python subprocess (3dphoto/main.py), so ComfyUI will look frozen while it works. Go make tea.

Inputs that matter

• img_path - and here's the first gotcha: it's an IMAGEPATH, not a standard image tensor. You can't wire an ordinary Load Image node into it. Use the pack's own LoadImagePath node (same category) to hand it a file, then chain PreViewVideo on the output.
• video_postfix - the camera move: dolly-zoom-in, zoom-in, circle, or swing. This is the knob you'll actually touch.
• traj_type - double-straight-line or circle; trajectory shape for the move.
• x_shift / y_shift / z_shift - camera translation. z_shift defaults to -0.05 (push in). Keep these small; the depth data only exists for the original view, and pushing too far is when artifacts creep in.
• fps (40) and num_frames (240) - that's a six-second clip by default.
• offscreen_rendering (true) - renders headless via Vispy's EGL backend.

Output: a single VIDEO. It wires into PreViewVideo.

The gotcha nobody warns you about

The output is a lie too. Look at the source and render() just returns an empty string for the VIDEO output - the actual file never gets passed down the graph. So the PreViewVideo preview will show you nothing, and you'll think it failed. It didn't. The MP4 lands in:

ComfyUI/custom_nodes/ComfyUI-3d-photo-inpainting/video/

Grab it from there. That's the real output path.

Install

ComfyUI Manager (search "3d-photo-inpainting"), or the old-fashioned way:

cd ComfyUI/custom_nodes
git clone https://github.com/AIFSH/ComfyUI-3d-photo-inpainting

Restart ComfyUI. On first import it auto-downloads the four PyTorch checkpoints (edge-model.pth, depth-model.pth, color-model.pth, model.pt) from the camenduru/3d-photo-inpainting Hugging Face repo into checkpoints/ - a few hundred MB, be patient. requirements.txt pulls in opencv-python, vispy, moviepy, transforms3d, networkx, scikit-image, matplotlib, and the __init__.py writes a 3dphoto.pth into your site-packages so the bundled research code is importable.

Troubleshooting

• "No module named 'MiDaS'" - the pack's own message: restart ComfyUI after the first import; the site-packages path was just written.
• Preview is blank - not broken, see the output gotcha above; check the video/ folder.
• Rendering fails or black frames - toggling offscreen_rendering to false uses a display-backed Vispy backend instead of EGL; one or the other usually works on any given machine.
• Tearing at the edges - you're moving the camera too far; back off the shifts.

If all you want is a gentle parallax, DepthFlow is smoother and faster. This pack's reason to exist is the disocclusion inpainting that lets the camera actually travel - a dolly zoom into a family photo is exactly the sweet spot. Just remember where the video lands.