Ideogram 4 Reference Conditioning

Ideogram 4 is the best open text renderer around, and it shipped with exactly zero inpainting support. Its single-stream architecture (text and image tokens in one shared sequence) has no masked-inpainting mode at all. So a community dev named BitPoet proved it could be taught - by training a LoRA on reference-image pairs and building this node to feed the reference image into the sampler at inference. This node is the inference half of that trick.

Ideogram 4 Reference Conditioning encodes one reference image at your target size and attaches its latent to your positive conditioning only. It's the missing plumbing that turns Ideogram 4 from "generate an image" into "edit this image." Alpha, one-person, and genuinely clever.

How it works

Ideogram 4 is a 9.3B, 34-layer DiT where text and image tokens run in one sequence. BitPoet's insight (announced on r/StableDiffusion in June 2026, alongside the Ideogram4-Inpaint-LoRA on Hugging Face) was that these unified models already know how to condition on a reference - you just need the latent handed to them the right way. He modified ai-toolkit to train LoRAs on image-reference-prompt triplets with loss computed only on the noisy latent, patched ComfyUI to load them, and the node follows the same contract at inference: text | noisy output | clean reference, with the reference at indicator 4, a relative MRoPE time coordinate of 1, and an Ideogram timestep of 1.0.

Concretely, the node takes your Empty Flux 2 Latent as the target, upscales the reference image to that size (bilinear), VAE-encodes it, and injects it into the conditioning via the reference_latents key. The sampler sees the clean reference alongside the noisy output - exactly the shape the LoRA was trained on.

The inputs that matter

• positive (CONDITIONING) - your Ideogram 4 positive conditioning.
• reference_image (IMAGE) - the image you want to edit or inpaint into.
• vae (VAE) - must be the Flux 2 / Ideogram 4 VAE. Feed anything else and the 128-channel check fails.
• target_latent (LATENT) - an Empty Flux 2 Latent, sized for your output.
• resize_mode (enum, default stretch) - stretch matches the ai-toolkit training preprocessing; center_crop keeps the aspect ratio.

Outputs: positive (CONDITIONING - this is the one you wire, and the only one: connect it to the Dual Model CFG Guider and leave the negative and unconditional Ideogram 4 conditioning untouched), reference_latent (LATENT - the encoded reference), and resized_reference (IMAGE - a preview of exactly what the model saw).

The workflow

The bundled workflow relies on Kijai's Ideogram4PromptBuilderKJ from KJNodes, which is how Ideogram 4 prompts anyway (it's a JSON-schema model, not a prose model). The inpainting trick rides on that: instead of drawing a mask, you put a bbox where the edit should happen and describe the change in the bbox's description. Don't fill in any other fields. Use "Default" or "Quality" mode - as the README puts it, "Turbo will mess up the image."

Installing it - read this twice

This is where the pack gets opinionated. The node pack is not enough, because stock ComfyUI doesn't support reference latents for Ideogram 4 yet:

# 1. The node pack
cd ComfyUI/custom_nodes
git clone https://github.com/BitPoet/ComfyUI-bitpoet-IG4Inpaint

# 2. The custom ComfyUI fork with the reference-latent support
git clone -b dev-ideogram4-inpaint https://github.com/BitPoet/ComfyUI.git

You also need the Ideogram 4 model itself (fp8 or nf4), the Flux 2 VAE, the LoRA (BitPoet/Ideogram4-Inpaint-LoRA on Hugging Face), KJNodes for the prompt builder, and you should copy workflows/idoinpaint_reference.png into ComfyUI's input folder (or load your own). The README calls the whole thing alpha - use at your own risk is not a disclaimer here, it's the tagline.

Troubleshooting

• ValueError: target latent must have 128 channels - you fed a non-Ideogram latent. Use an Empty Flux 2 Latent.
• Node errors on load or the conditioning key is ignored - you're on stock ComfyUI. The fork is mandatory.
• Reference latent shape mismatch - keep the Empty Flux 2 Latent fixed and let the node resize the reference to it, not the other way around.
• The edit looks mushy - this LoRA was trained on a hand-made 24-image dataset with checkpoints at 4000–5000 steps. Promising, not polished. Manage expectations.

And the licence: Ideogram 4 is non-commercial. This is a fun research toy and a proof of concept; it's not a commercial inpainting pipeline.