Lotus

If you've ever wired up a depth ControlNet, you know the drill: an auxiliary node pack, a 400MB download, a preprocessor that takes six seconds, and a map that looks like grey mush. Lotus is the other kind of preprocessor. It's a 1.7GB self-contained model that runs a single forward pass per image, and this node wraps the whole thing so you connect one wire and get a map back.

What it actually is

Lotus is a monocular geometry estimator - disparity depth in one mode, surface normals in the other. It came out of the EnVision-Research line and the SD2.1 "Lotus-G" checkpoint is the one that showed up in the big community shootout of eight depth models, where people liked its contrast and object separation and grumbled that its depth ordering wasn't always right (the classic complaint: it put a foreground pipe closer than the man holding it). Keep that trade in mind - Lotus is crisp, not infallible.

Where you'd reach for it:

• Depth ControlNet conditioning, especially at weight ~0.5-0.7 to set composition and let the sampler handle detail.
• Depth-aware post-processing: bokeh, atmospheric haze, background defocus.
• Parallax and Displacement tools (DepthFlow and friends), where a sharp map beats a smooth one.
• Video batches. At ~1.5s an image in fp16, you can afford to map 100 frames, which you can't with Marigold.

How it works under the hood

The mechanism is unusually clean. The node builds a UNet2DConditionModel from a config JSON shipped in the pack, reading in_channels straight off the checkpoint. Your image gets VAE-encoded to a latent, the UNet runs once at a fixed timestep of 999 with the task - depth or normal - injected as a class embedding, and the output latent is decoded and bilinearly resized back to your original resolution.

Two details explain the pack's whole pitch. There's no CLIP: instead of dragging a T5/CLIP loader in to feed an empty string, the node loads a 2MB precomputed empty-prompt embedding from configs/. And depth/normal are separate checkpoints, chosen by mode, which is why flipping modes triggers another download the first time.

The inputs that matter

image goes in, one IMAGE comes out. On the required settings:

mode - depth or normal. Depth is the ControlNet workhorse; normal gives you RGB surface-direction vectors for relighting and PBR-ish work. Worth knowing: normal-map ControlNet is one of the conditions that never got rebuilt for post-Flux models, so you're likelier to feed this into a relight node than a ControlNet.

resolution - internal processing size, 256 to 2048 in steps of 64, default 1024. The output is always resized back to your input size, so this is purely a speed/quality knob. Drop it to 768 or 512 for video; the author says 2-3x faster.

colormap - default gray, which is what ControlNet wants. spectral, turbo and inferno are for looking at, not conditioning. normalization (min_max vs raw) only matters if you're feeding raw values somewhere else - min_max stretches each frame to 0-1 using that frame's own min and max, so on a video you'll see the contrast breathe frame to frame. It's a rendering nit, not a bug.

weight_dtype defaults to fp16 and is fine. If a map comes back black or speckled, switch to bf16 before blaming the model - that's the usual fp16 overflow. unload_model clears VRAM after the run, which helps if a big sampler comes next, but it also drops the cached UNet, so the following run pays the load again.

Install

ComfyUI Manager, search ComfyUI-Lotus-2, or by hand:

cd ComfyUI/custom_nodes
git clone https://github.com/1038lab/ComfyUI-Lotus-2
pip install -r ComfyUI-Lotus-2/requirements.txt

Heads up: the README's own clone line says ComfyUI-Lotus2.git - use the URL you actually landed on. And that pip install matters. The README claims "zero external bloat", but requirements.txt is diffusers, and this node imports diffusers.models directly. Install it manually and skip the deps, and the node fails to load.

Weights download themselves into models/geometry_estimation/ on first run, 1.73GB per mode. The node also needs a Stable Diffusion VAE: it looks in models/vae for a name containing ft-mse, sd-vae or sd2, and if it finds nothing it takes any .safetensors sitting there. If your VAE folder is all SDXL and Flux, keep vae-ft-mse-840000-ema-pruned.safetensors around - otherwise check the console for the [Lotus] Using local VAE: line and confirm it picked something sane.

Where people get burned

Video flicker is the big one. Lotus is an image model looped over frames, so temporal stability is whatever your frames give it, and per-frame min-max normalization can make a moving shot pulse. If you need stable depth across a clip, that's a video-native model's job, not this one. Second: if your workflow depends on precise depth ordering rather than crisp edges, test before you commit - Depth Anything V2 Large is still the safer default for straight conditioning. Lotus is the pick when you want detail, speed, and normal maps from the same node.