X -> RGB Inpainting

Plain X2RGB regenerates a whole image from material maps. This node does the same thing but only inside a mask - you give it a reference photo, a mask of the region you want to regenerate, and the material maps that should guide that region. The classic use: relight one object or one shadowed corner of a photo while keeping the rest of the frame exactly as-is, instead of regenerating everything and hoping the parts you liked survive.

This is the third of the three RGB-X pipelines from the research project (the original ComfyUI wrapper only exposed one, which is half the reason this pack exists). If you already understand X2RGB, this is that, plus a mask, plus three outputs instead of one.

How it works

The node takes your photo, converts it sRGB→linear, and uses your mask to build the masked image the inpainting pipeline samples around. One detail the author handled for you: ComfyUI's MASK convention is white = the area to inpaint, while the underlying pipeline expects the opposite, so the node inverts it internally. Feed it a standard white-on-black ComfyUI mask and don't overthink it.

All five material maps are optional, same preprocessing as X2RGB (albedo and irradiance go linear, normals get normalized, roughness/metallic get clamped), plus an optional prompt. The pipeline runs a DDIM pass with the same guidance_scale / image_guidance_scale pair, and because you fed a photo, you also get the photo itself back in VAE-decoded form.

The three outputs

• generated - the one you actually want. The photo with the masked region regenerated from your maps.
• masked_photo - the masked photo as the model saw it (VAE-decoded), i.e. the reference with the inpaint region zeroed out. Handy for diagnosing "why is it generating a hole here."
• photo_vae - the reference photo run through the VAE. Mostly diagnostic too, but useful if you're compositing and need everything in the same decoded space.

For your own sanity: generated is what goes to a SaveImage. The other two are inspection tools, not results.

The inputs that matter

• photo - reference image. Must be an IMAGE.
• mask - a MASK, where white = the region to regenerate. Invert it if you only know how to paint "keep" masks.
• seed / steps / guidance_scale / image_guidance_scale - same defaults as X2RGB (0 / 50 / 7.5 / 1.5); image guidance is again the knob that decides whether your maps actually win inside the mask.
• albedo / normal / roughness / metallic / irradiance + prompt - all optional. The natural move is extracting these from the same photo with RGB2X, editing the ones you want changed, and feeding them back in - that's a full material-constrained relight of one region.

Outputs: generated, masked_photo, photo_vae (all IMAGE).

Install and models

ComfyUI Manager (search "RGB-X") or git clone https://github.com/leob03/ComfyUI-rgbx.git into custom_nodes/, restart. Dependencies are diffusers>=0.20.0, transformers, torchvision - normally already there. This pipeline's weights live in ComfyUI/models/rgbx/x-to-rgb-inpainting/:

huggingface-cli download zheng95z/x-to-rgb-inpainting --include "*.safetensors" --local-dir models/rgbx/x-to-rgb-inpainting

Skip the download and the node tries HuggingFace at first run instead. ~12GB VRAM, fp16.

Where people get burned

The resolution of the output follows your first non-None material map, not the photo - feed maps at the photo's resolution or the inpainted region will be regenerated at a different scale and get stretched on composite. And the mask convention trip (white = inpaint) is worth repeating because it's backwards from what most people expect from other inpainting nodes. Keep the mask edge soft if you can; hard masks make the boundary obvious on relight.