ComfyUI Ideogram 4 Regional LoRA

ComfyUI Ideogram 4 Regional LoRA

Apply a LoRA by bounding box in Ideogram 4 — give each character in the frame its own LoRA, in a single forward pass, on the MODEL path.

For the local Ideogram 4 DiT (the ideogram4_*.safetensors diffusion models, not the Ideogram cloud API nodes).

🧍🧍 The node

| | | |---|---| | Category | Ideogram4 | | Node | Ideogram 4 Regional LoRA | | Models | Ideogram 4 (local weights, e.g. ideogram4_fp8_scaled) |

Why this exists

Ideogram 4's JSON prompt carries a bbox per element, but those coordinates are just text tokens — the model has no spatial conditioning to hang a per-region LoRA on. So loading two character LoRAs the normal way merges both into the weights globally and every face gets the average of the two identities.

This node builds the missing spatial handle out of the model's own sequence layout. Ideogram 4 runs one packed [text | image] sequence where the image tokens are a row-major (gh, gw) grid, so a bbox maps exactly onto a set of token positions. Each LoRA's low-rank delta is then applied in activation space and weighted per token:

out = base_out + w_token · B(A(x))

Base weights everywhere else, one model pass, nothing baked into the weights. The delta is added by a forward hook on each target Linear, which leaves the module and its .weight path intact so ComfyUI's own weight-patch and fp8 weight-casting stay undisturbed.

Wiring

Ideogram 4 Prompt Builder KJ ──┬─ bboxes ─┐
                              ├─ width  ─┤
                              └─ height ─┤
UNETLoader (ideogram4)         ── model ──┤   Ideogram 4     ├─ model ────────► DualModelGuider
UNETLoader (…unconditional)  ── model_uncond ┤ Regional LoRA ├─ model_uncond ──►

• Starts with one LoRA slot and a + Add LoRA button. Each slot is three widgets — which LoRA, how strongly, which region — and a slot left at None does nothing, so there is no separate enable toggle. - Remove last takes one away (and parks it at None, so a hidden slot can never still apply). Both actions are also on the node's right-click menu. Up to 8 slots.
• region_index_N counts placed boxes in editor order, so index 0 is the first region you drew. Both region sources (bboxes, or the caption prompt string) are indexed the same way, and both skip unplaced elements.
• Ideogram 4 does CFG with two models, so wire the unconditional one through model_uncond too — the workflow's own "load the same LoRA in both" rule applies. Leaving it unwired keeps the uncond at base weights, which makes CFG pull the region back toward base identity: a legitimate A/B (it amplifies the delta by the CFG factor), not the default.
• Don't also load these LoRAs in a Power Lora Loader — that applies them globally and the masking would be moot.
• The three tuning knobs sit after the slots, not interleaved with them: feather_cells softens every region edge in latent cells (1 cell ≈ 16 px); overlap_mode decides what happens where two bboxes intersect (normalize scales down to sum 1, sum stacks, first_wins lets the earlier slot occlude); mask_adaln is discussed below.

What to expect

Measured on ideogram4_fp8_scaled with a rank-32 face LoRA at strength 1.0, single forward, half-image bbox (a second face LoRA gave the same picture):

| | | |---|---| | Full-image bbox vs a stock global LoRA | 4.0% of signal (effect size 37.5%) | | Off-target change vs a global LoRA | 3.0× less | | In-bbox strength vs a global LoRA | ~88% of its magnitude |

The residual on the first row is bake-vs-bypass on quantized weights: the stock path bakes the delta into an fp8 weight, this node adds it in bf16 activation space. In fp32 the two agree to 0.2% (see tests/test_delta.py).

Two caveats worth knowing:

• Attention crosses the seam by design. In-bbox tokens exchange keys and values with the rest of the image at every one of the 34 layers, so the non-target region does move a little — the delta is concentrated in the bbox, never perfectly confined to it. The leakage is diffuse rather than a halo at the boundary, so feathering softens the edge but won't remove it.
• adaln_modulation is broadcast, not masked at the Linear. That projection only sees the per-batch timestep embedding, so its delta is weighted by the full-sequence mask instead — base modulation outside the bbox, LoRA-shifted inside. Including it matters: those 34 modules carry 19–26% of a face LoRA's total effect across the two measured, so mask_adaln defaults on. Turn it off to save the transient per-token modulation tensor (~150 MB/block at 4k tokens).

Compatibility

Safe to chain after Ideogram4OptimizationsKJ. Its chunk_ffn option splits the feed-forward over the token dim, so hooks on feed_forward.w1/w2/w3 see consecutive slices instead of the whole sequence; the node tracks a per-module token offset to reassemble the right mask slice. If the layout can't be resolved for any reason the node fails closed to base weights rather than silently applying a character LoRA to the whole image.

🔢 Ideogram 4 Region Preview

Answers "which element is index 1?" by drawing it.

| | | |---|---| | Category | Ideogram4 | | Node | Ideogram 4 Region Preview |

Feed it the same JSON string the text encoder gets. It labels every placed bbox with the exact region_index the LoRA node expects — zero-based, counting only elements that carry a bbox — and returns both the image and a text report listing each index with its pixel extent and description. Wire the decoded image into image to see the regions sitting on the render they produced; leave it unwired to draw on a blank canvas, which costs nothing to run.

This numbering is not the prompt builder's numbering. Ideogram4PromptBuilderKJ's own preview tags boxes 01, 02, … — one-based, and counting unplaced elements too. Typing one of those into region_index selects the wrong region, silently. That mismatch is the reason this node exists.

The dashed outline shows the feather_cells reach, so you can see how far the soft edge actually extends before committing to a render.

To adjust boxes by dragging rather than editing coordinates: wire the JSON into Ideogram 4 Prompt Builder KJ's import_json with import_mode = "when empty" (it seeds the editor once, then the editor wins), drag on its canvas, and take its prompt output as the new source.

Example workflow

examples/IDEOGRAM4_REGIONAL_LORA_demo.json is a face + hands demo: a face LoRA masked to the face bbox, a hands LoRA to the hands bbox, both at strength 1.0. It mirrors a standard Ideogram 4 generation path but replaces the usual pair of global LoRA loaders — where two such LoRAs have to be throttled to roughly 0.2 each to keep out of each other's way — with one regional node. Point the two lora_* slots at your own files and edit the JSON prompt node. examples/build_workflow.py regenerates the workflow.

Note each LoRA's trigger word belongs only in the element that LoRA is masked to, which is the thing global loading makes impossible.

Tests

python tests/test_regions.py   # bbox parsing + rasterizer math, needs only torch
python tests/test_delta.py     # delta path vs real comfy adapters, needs comfy importable
node   tests/test_slots_ui.js  # slot add/remove logic, ComfyUI globals stubbed

How the growing slot list works

The Python node declares 8 fixed slots so ComfyUI's ordinary widget serialization keeps working; web/js/regional_lora_slots.js only decides how many are on screen. Nothing about it changes what the backend receives — a slot whose lora_N is None is skipped there whether it is visible or not — so if the frontend extension ever fails to load, the node still works, it just renders all 8 slots at once.

The row count is not stored in the graph. On load it is inferred from the highest slot holding a real LoRA, so a saved workflow reopens showing exactly the slots that do something. The two buttons are created with serialize: false, which the frontend honours on both save and load, so they never occupy a widgets_values position.

Install

cd ComfyUI/custom_nodes
git clone https://github.com/kat3ri/ComfyUI-ideogram4-regional-lora
# restart ComfyUI

Requires a ComfyUI new enough to include comfy/ldm/ideogram4/ (local Ideogram 4 support). No extra Python dependencies.