Load DiLight ControlNet

What this actually is

This is the loader half of a two-node pair - Load DiLight ControlNet plus Prepare DiLight CN Image - that runs DiLightNet inside ComfyUI. DiLightNet is a SIGGRAPH 2024 paper ("Fine-grained Lighting Control for Diffusion-based Image Generation") that re-lights a foreground object to match a lighting environment you hand it as rendered shading images. It's built on Stable Diffusion 2.1, and this node is what gets its weights into your graph.

Why you'd reach for it: DiLightNet is one of the few diffusion methods where lighting is the conditioning signal itself, not a prompt adjective. IC-Light and friends take a lighting description as text; DiLightNet takes a rendered shading hint and reproduces that exact lighting on the subject. The catch is those hints are images you have to render yourself, which is the real story of this whole pack.

Why it isn't the normal ControlNet loader

At a glance this is just a ControlNetLoader with one dropdown. It is not. A stock controlnet's input-hint block is built for a 3-channel RGB condition - canny maps, depth maps, that world. DiLightNet's conditioning is a 16-channel tensor: a reference image, a mask, and 12 channels of shading. The loader does the work ComfyUI can't guess:

1. It calls ComfyUI's built-in ControlNetLoader to load the DiLightNet checkpoint as a standard ControlNet.
2. It strips every controlnet_cond_embedding.* key out of the file and loads them into NeuralTextureEmbedding - the same conditioning block from the DiLightNet paper.
3. It swaps that block in as the controlnet's input-hint block (subclassing the model so the block follows along on .to(device)), and pins it to fp16.

Net effect: a CONTROL_NET object whose hint encoder accepts 16 channels instead of 3. You pick a file and it just works.

The inputs and outputs that matter

There's exactly one input: control_net_name, the dropdown listing every .safetensors in your ComfyUI/models/controlnet/ folder. Select the DiLightNet file you downloaded (below). That's the whole UI.

Output: CONTROL_NET, wired into the control_net slot of ControlNetApplyAdvanced. The same node's image slot takes the 16-channel image from Prepare DiLight CN Image, positive/negative come from your CLIP encodes, and the result feeds KSampler. The pack ships a working reference (example_workflows/dilight_example_workflow.json) showing the full graph.

Installing it

Via ComfyUI Manager, search ComfyUI-DiLightNet and install. Or:

cd ComfyUI/custom_nodes
git clone https://github.com/logtd/ComfyUI-DiLightNet

Restart ComfyUI after. There's no requirements.txt - the pack leans entirely on ComfyUI's own torch, so no dependency hell for once. But you do need three model files:

• The DiLightNet controlnet: diffusion_pytorch_model.safetensors from the dilightnet/DiLightNet HuggingFace repo, into ComfyUI/models/controlnet/ (the example names it dilight.safetensors).
• Stable Diffusion 2.1's UNet and its OpenCLIP text encoder - the README points at stabilityai/stable-diffusion-2. This model is SD 2.1-only; don't try it on a 1.5 or SDXL checkpoint.

Where people get burned

• Pointing the loader at the wrong file. The node doesn't validate before load_state_dict; feed it a normal controlnet and it dies with a missing/unexpected-key error. Give the DiLightNet file a distinct name so it never gets confused with your canny model.
• Expecting it to work alone. Without Prepare DiLight CN Image there's no valid 16-channel hint to feed it - a plain LoadImage output is 3 channels and the controlnet will choke. The two nodes are a pair.
• The radiance hints themselves. This pack deliberately omits the Blender/Dust3r code that renders the shading images; the README says those are destined for separate repos. No rendered diffuse + GGX passes, no condition, and no amount of loader fiddling fixes that. It's the genuine barrier to entry.

It's a quiet corner of the ecosystem - essentially no Reddit footprint, and the upstream project expects a Python 3.10 + Blender environment of its own. But if you've gone to the trouble of rendering shading hints and want them inside a ComfyUI graph, this is the node that makes it possible.