DepthDiff

Regular img2img gives you one global dial: a denoise value that applies to the whole frame. Crank it and your subject changes but so does your background; drop it and nothing moves. Differential Diffusion is the upgrade - every pixel gets its own dial, so you can keep the sky while repainting the ground, or swap an outfit while leaving the face alone. DepthDiff is the whole technique squeezed into a single ComfyUI node.

What it actually does

The mask is everything. DepthDiff builds a grayscale mask from your input image, then uses it two ways at once: it patches the model and it attaches the mask to the latent. Where the mask is bright, the sampler re-noises that region hard; where it's dark, those pixels are essentially frozen. Default is invert = true, so dark regions get more diffusion, bright regions get preserved - flip it to reverse. The name isn't the full story: by default the mask comes from luma (brightness), which is great for relighting-style edits where you keep the highlights. Turn on depth_mode and it runs Depth Anything V2 first, so the mask comes from scene geometry instead - near objects re-drawn, far objects kept.

How the mechanism works

Differential Diffusion (Chen et al., arXiv 2306.00950 - the "give each pixel its strength" paper) runs entirely at inference, no training. DepthDiff implements it via ComfyUI's set_model_denoise_mask_function patch. Each sampling step, the patch computes how far through denoising you are and re-noises only pixels where mask × multiplier is above that threshold. Early steps have a high threshold, so only the strongest mask regions get touched; as sampling progresses the threshold falls and more of the image gradually comes in. Net effect: mask value = edit strength, from "don't touch" to "full renoise." That's the difference between a hard binary inpainting mask and the soft, gradient-driven result that makes seamless edits work.

The node is unusually self-contained. It VAE-encodes your image internally, so there's no separate Encode node. In depth_mode it downsizes to depth_max_size (default 1024 long side) for the depth pass, then upscales the map back - a nice touch that keeps VRAM sane. It even saves a preview of the mask and shows it in the UI, a godsend when you're dialing in levels.

Inputs that matter

You feed it model, image, and vae - that's it for the plumbing. The widgets you'll actually touch:

• invert - defaults true. Dark = more diffusion. Get this backwards and you'll wonder why your edits target the wrong thing.
• input_black / input_white - levels remap on the mask, in 0–255. Your first move when the mask is too washed out or too dark.
• gamma, brightness, contrast - tone shaping. Brightness is additive, gamma is a power curve; gamma is usually the one that feels right.
• strength - clips the whole mask down; effectively a global "how aggressive" knob.
• diff_diffusion_multiplier - a second global dial, 1.0 default, >1 more aggressive, <1 more preservation. Think strength = mask gain, this = sampler-side throttle.
• The optional mask input gates the whole thing - handy if you want depth-driven diffusion only inside a region you already have.

Outputs

All three matter: model (the patched model), latent (with the noise mask attached), and mask (the raw mask, in case you want to feed it somewhere else). Wire model and latent into your KSampler and you're done.

Install

Via ComfyUI Manager: search DepthDiff and hit Install. Or:

cd ComfyUI/custom_nodes
git clone https://github.com/spiritform/Comfy-DepthDiff.git

Then restart ComfyUI. The core node has zero Python dependencies - it runs out of the box. The catch: depth_mode requires Kijai's comfyui-depthanythingv2 pack too (Manager: search "DepthAnythingV2"), and the V2 weights auto-download to ComfyUI/models/depthanything/ on first use. The dropdown offers VIT-S/B/L plus the Giant, each in fp16/fp32. To just test, skip depth_mode entirely: feed the node any image (a precomputed depth map or a luma render works) and it never touches the depth dependency. Stick with VIT-L, the community daily driver, unless you have headroom - the Giant is the quality ceiling but it's slow and OOMs on modest VRAM.

Where people get burned

Running depth_mode without Kijai's pack raises a clear RuntimeError - it won't silently degrade, so you'll know. The subtler trap is invert. On a photo with a bright subject on a dark background, the defaults preserve the subject and re-paint the background - often exactly backwards for a portrait edit. Check the preview mask before you burn a batch. And remember this is a soft edit by design: near-black regions aren't untouched, just barely touched. If you want a hard boundary, drop the blur to 0 and use the optional mask input as the gate.