Mpi Lift Depth

Somewhere in your render there are holes - pixels no geometry was drawn for, which a model painted in afterwards. The fill looks fine in the image. It has no depth. The moment anything downstream needs the scene's geometry (a second camera move, a re-render from another angle, a 360 path) that flat patch is a void in your depth buffer.

MpiLiftDepth shoves a painted fill back into the scene: run MoGe on the fill, fit its relative depth to the camera z the scene already knows, write out a raw float32 depth file kept only on the holes.

How it works

The depth model is MoGe v1 - Microsoft Research's monocular geometry estimator, MIT-licensed, vendored into scene3d/moge/ with the licence beside it - which predicts a point map rather than a grayscale depth map, and is scale-and-shift agnostic. That's precisely the ambiguity this node resolves: MoGe gives you relative depth over the whole image, and your known_depth file gives you true camera z where the render actually drew something. A least-squares line z = a·zm + b is fitted on those known pixels - eroded 9px off the hole boundary first, so the leak at the edge of the fill doesn't drag the line - and then applied to the fill. The fit is kept only on the holes (grown 5px), minus depth edges, where the 3×3 neighbourhood spans more than 5% of the depth. You don't want the fill's silhouette introduced as a cliff.

The guards are the interesting part. It fits scale only - a = the median ratio, no shift - when the known pixels sit at effectively one depth, when the least-squares slope comes out ≤ 0, or when the shift comes out negative on a frame with no back-faced known pixels, which is the outside case where a negative shift pulls the near fill toward the camera. Inside a house, the shift is the room's own fit and stays.

Inputs and outputs

• image - the fill, i.e. a render whose holes a model painted.
• known_depth - a raw float32 file in input/, same size as the image, row-major: camera z per pixel. 0 = unknown (the hole, kept). Positive = known (used for the fit, not kept - those are the render's own pixels). Negative = known at its size for the fit AND kept: a surface the fill replaces. That case is why the node exists: a camera inside a house sees the backs of walls - holes the fill must cover, but also the only thing in frame the room can be fitted to. Fitted on the view out alone, the room landed ~8% past the walls and hid behind them.
• fov_x - the render's horizontal field of view in degrees, default 60 - the pinhole the ray maths is built from.
• model - a MoGe .safetensors from models/moge/.
• ground (optional) - the ground plane in the render's camera frame (x right, y down, z ahead) as 'nx,ny,nz,d'. Any kept pixel the fit puts under that plane moves onto it. Empty means no ground. The author's caveat is honest: a real pit under the ground flattens too.

Outputs are depth_path (a STRING) and fit_error (a FLOAT, the median relative error on the known pixels). The file lands under output/scenes/ as raw little-endian float32, rows × columns, no header - for a host app or your own script, not a Preview Image. fit_error is your sanity check: read it, don't ignore it.

Installing it

cd ComfyUI/custom_nodes
git clone https://github.com/MadPonyInteractive/ComfyUi-MpiNodes
# restart ComfyUI

Then the weights, which you place yourself - the pack never downloads anything. Drop a MoGe v1 .safetensors into ComfyUI/models/moge/, which registers as a model folder the combo reads. The config hardcoded in the source is the ViT-L one (Ruicheng/moge-vitl re-saved as safetensors), so that's the checkpoint to grab. The pack has no requirements file - it borrows scipy, opencv and torchaudio from the ComfyUI environment, and those imports aren't lazy.

Where people get burned

The raises are clear when they happen: the known_depth file isn't in input/, it doesn't hold exactly w × h values, or fewer than 2 known pixels survive to fit against - usually meaning the map is empty or belongs to a different render.

The subtle one is the sign convention: a positive known value is not kept. Write a map where everything is positive because "that's where I know depth" and you get a fit and an empty fill. Zero is the hole.

MoGe sits on the CPU between runs (~1.3 GB of RAM), so this node is a permanent RAM tenant even when idle. It also hashes the depth file it reads, so overwriting a map in place re-triggers instead of serving a cached result.