AOV to RGB

The name sounds like a file format, but what x2rgb ("AOV to RGB") actually does is closer to magic for anyone who renders. Feed it the albedo, normal, roughness, metallic and lighting passes out of your 3D app, and it synthesizes a photorealistic image of the scene. No renderer, no GI bake, no waiting on the GPU farm - a diffusion model looks at your passes and paints the final frame, guided by whatever text you give it.

This is the reverse direction of the RGB↔X project, a real SIGGRAPH 2024 paper from Adobe Research (Zheng Zeng and friends), wrapped into ComfyUI by flrngel. The repo is a fork of toyxyz's earlier ComfyUI_rgbx_Wrapper, and both halves ship here: rgb2x guesses AOVs from a photo, x2rgb goes the other way. If you've ever seen the RGB-X name in a specular-map thread and wondered what the fuss was, this is it.

How it works

Under the hood it's a Stable Diffusion model (zheng95z/x-to-rgb, auto-downloaded from Hugging Face) with an "AOV dropout" trick. The model was trained so you can hand it some of the channels and it will faithfully respect those while hallucinating the rest. Hand it albedo + normal and it fills in plausible roughness and lighting. Hand it all five and it locks the image down to what a render of those passes would look like - the "middle ground between rendering and generation" the paper sells.

The wrapper does the fiddly color-space work for you, which is easy to forget until you try this yourself: albedo and irradiance get gamma-corrected from sRGB to linear, the normal map is rescaled to [-1, 1], roughness and metallic just get clamped. Then it runs a DDIM scheduler with zero-SNR / trailing timestep settings, which is why the defaults are what they are.

The inputs that matter

You don't set much. prompt defaults to "masterpiece, best quality, photorealistic" and behaves like any SD prompt - for interior scenes (this model's home turf) it mostly steers style and mood. seed and steps (50) are the usual. Two knobs actually matter:

• guidance_scale (7.5) - standard CFG for how hard the text pushes the result.
• image_guidance_scale (1.5) - the interesting one. This is effectively your "how strictly do I follow the AOVs" dial. Crank it up and the output hews to your passes; lower it and the model wanders. This is the setting you'll actually tune.

The five optional inputs are albedo, normal, roughness, metallic, and irradiance - all IMAGE tensors. You can leave any of them disconnected (the dropout is the whole point), but at least one is required. Wire the output IMAGE into a Save Image or preview node like any other generator.

Installing it

Easiest via ComfyUI Manager - search "ComfyUI_rgbx_xrgb_Wrapper". Or the manual route:

cd ComfyUI/custom_nodes
git clone https://github.com/flrngel/ComfyUI_rgbx_xrgb_Wrapper

Then restart ComfyUI. Dependencies are the usual suspects - torch, diffusers, transformers, opencv-python, imageio, huggingface-hub - most of which you already have. The real gotcha is the first run: the model weights for zheng95z/x-to-rgb download automatically into the pack's model_cache folder, which is a chunky multi-GB download that can look like the node is frozen. Watch the terminal. And the original project recommends a GPU with at least 12GB VRAM - this is fp16 but it's still a full SD UNet plus a VAE.

Gotchas

• The author says it plainly in the README: "Vibe coded. I won't maintain this repo." This is a research wrapper, not a polished product. It works, but don't file issues expecting fixes.
• It's trained for interior scenes. Feeding it a face or a landscape is asking the model to do something it wasn't built for - expect weird results.
• Feed it LDR images (PNG/JPG). ComfyUI hands it regular IMAGE tensors and the node assumes sRGB inputs it then linearizes internally. Don't bother trying to sneak EXR in.
• No AOVs to start with? Run the sibling rgb2x node on a photo first, then chain its outputs into x2rgb to relight or restyle.