ComfyUI Marigold Intrinsics PaGeR

This custom node pack provides Marigold IID nodes and PaGeR ERP geometry nodes.

Included Nodes

TODO: missing PaGeR-Normals PaGeR models 5/6

Marigold IID Loaders

1. Load Marigold IID Appearance Model (DownloadAndLoadMarigoldIIDAppearanceModel)

• Downloads and loads the Marigold appearance pipeline as IIDMODEL.
• Uses hardcoded in-memory empty conditioning (no precomputed .safetensors path).

2. Load Marigold IID Lighting Model (DownloadAndLoadMarigoldIIDLightingModel)

• Downloads and loads the Marigold lighting pipeline as IIDMODEL.
• Uses hardcoded in-memory empty conditioning (no precomputed .safetensors path).

3. Load Marigold IID Split (UNet/CLIP/VAE) (MarigoldIIDSplitLoader)

• Loads split components as Comfy MODEL, CLIP, VAE.
• Uses runtime-minimal base files by default (scheduler + VAE); text assets are only fetched if load_comfy_model_clip is enabled.
• Supports tiny VAEs, including TAESD-style VAE choices.

4. Translate Marigold Split -> IIDMODEL (MarigoldIIDSplitToIIDModel)

• Converts split-loaded MODEL + CLIP + VAE into IIDMODEL.
• Uses hardcoded in-memory empty conditioning and TAESD-compatible VAE adapters.

Marigold IID Inference

1. Marigold IID Appearance (MarigoldIIDAppearance)

• Runs appearance inference and returns albedo, roughness, metallicity.

2. Marigold IID Appearance (with Material) (MarigoldIIDAppearanceExtended)

• Same as above, plus material.

3. Marigold IID Lighting (HyperSim) (MarigoldIIDLighting)

• Runs lighting inference and returns albedo, shading, residual.

PaGeR

1. Load PaGeR Model (DownloadAndLoadPaGeRModel)

• Loads the three new DA3 checkpoints or an existing legacy PaGeR depth/normal model.
• New nodes default to DA3 / Depth Anything 3; legacy model selections remain routed to the unchanged SD2.1 loader.
• Legacy SD2.1 models use hardcoded in-memory empty conditioning and support the optional Comfy TAESD VAE backend.
• DA3 models do not use the legacy diffusion/UNet/VAE/empty-conditioning path.

2. PaGeR Infer Cubemap (ERP) (PaGeRInferCubemap)

• Node A in the PaGeR pipeline.
• Runs encode + UNet + decode and returns decoded cubemap predictions.

3. PaGeR Depth Postprocess (ERP) (PaGeRDepthPostprocess)

• Node B depth postprocess.
• Converts cubemap prediction to ERP outputs:
  • depth
  • edge_mask
  • depth_color

4. PaGeR Normal Postprocess (ERP) (PaGeRNormalPostprocess)

• Node B normal postprocess.
• Converts cubemap prediction to ERP normal visualization.

5. PaGeR Save Point Cloud (GLB/PLY) (PaGeRSavePointCloud)

• Exports point cloud from color + depth as .glb or .ply.
• Supports depth_mode auto logic, optional masks, and optional edge filtering.

PaGeR DA3

The DA3 path supports prs-eth/PaGeR, prs-eth/PaGeR-metric-depth, and prs-eth/PaGeR-normals. It is separate from the legacy SD2.1 cubemap pipeline. Released DA3 checkpoints auto-download into ComfyUI/models/marigold_intrinsics/pager_da3/<repo-id>/.

1. PaGeR DA3 Runtime runs the selected DA3 checkpoint and returns a modality-aware result.
2. PaGeR DA3 Save Images saves raw TIFF maps and PNG previews.
3. PaGeR DA3 Save NPZ saves evaluation arrays and scene/model metadata.
4. PaGeR DA3 Save Point Cloud exports metric-depth point clouds as GLB or PLY.

The unified checkpoint requires an explicit indoor or outdoor selection. Automatic CLIP scene classification is intentionally not included, so DA3 inference does not load or run another model.

Recommended PaGeR Pipeline

For depth:

1. Load PaGeR Model with a depth checkpoint.
2. PaGeR Infer Cubemap (ERP).
3. PaGeR Depth Postprocess (ERP).
4. Optional: PaGeR Save Point Cloud (GLB/PLY).

For normals:

1. Load PaGeR Model with a normals checkpoint.
2. PaGeR Infer Cubemap (ERP).
3. PaGeR Normal Postprocess (ERP).

Hardcoded Empty Conditioning

The Marigold and legacy SD2.1 PaGeR loaders use hardcoded in-memory zero embeddings for empty conditioning:

1. No .safetensors empty-conditioning file is required.
2. No precomputed-conditioning inputs are exposed on loader nodes.
3. Marigold uses shape [1, 2, C], where C is inferred from UNet cross_attention_dim.
4. Legacy PaGeR uses shape [1, T, C], where T defaults to SD2-style context length and C is inferred from UNet config.
5. DA3 PaGeR does not use empty conditioning.

TAESD VAE Option

Load PaGeR Model supports an optional Comfy TAESD VAE backend.

Why use it:

1. Lower VRAM usage.
2. Faster VAE encode/decode on constrained GPUs.
3. Useful when fitting PaGeR workflows on smaller cards.

Tradeoff:

1. TAESD is a tiny/approximate VAE.
2. Output quality can degrade in fine details, smooth gradients, and sharp depth edges.
3. In many scenes it is still usable, especially when speed and memory are priority.

8GB VRAM Note

This setup has been tested on an 8GB GPU with practical settings:

1. Prefer fp16 precision.
2. Use hardcoded in-memory empty conditioning (default).
3. Enable TAESD VAE when memory is tight.
4. Keep PaGeR as Infer Cubemap -> Postprocess (current default workflow).

Dependencies

Install with:

pip install -r requirements.txt

Citation

Please cite our paper: (Waiting for citation)

Put citations here

License

This code of this work is licensed under the Apache License, Version 2.0 (as defined in the LICENSE).

The models are licensed under RAIL++-M License (as defined in the LICENSE-MODEL)

By downloading and using the code and model you agree to the terms in LICENSE and LICENSE-MODEL respectively.

Acknowledgements

This project builds upon and is inspired by the following repositories and works:

• Marigold-e2e-ft, based on paper Fine-Tuning Image-Conditional Diffusion Models is Easier than You Think.
• Marigold, based on paper Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation.

We thank the authors and maintainers for making their code publicly available.