GRAG Patch (Group-Relative Attention Guidance)

Every instruction editor makes you argue with the same trade-off. Ask Qwen-Image-Edit 2511 to put a hat on someone and you get a near-copy of your input with a slightly different hat strap; push it harder and it decides now is the moment to relight the room and give everyone new faces. There's a real dial for that trade-off, and almost nobody's workflow has one. GRAG Patch is it.

What it actually does

It's a MODEL patch, nothing more. Load your Qwen-Image-Edit checkpoint as usual, run the MODEL wire through this node, then into your sampler. No weights replaced, no VAE or CLIP swap, no sampler trickery - the diffusion model stays exactly what you loaded.

Under the hood it implements GRAG from arXiv 2510.24657, a port of little-misfit's reference implementation onto comfy's native DiT stack. In a joint text+image attention block, each key splits into two parts: a shared, layer-only bias (just "how this layer behaves when it edits anything") and a per-token deviation carrying the content-specific edit signal. GRAG reweights those separately, after RoPE, on the keys only:

k' = cond_b * mean(k_ref) + (k_ref - mean(k_ref)) * cond_delta

Flatten cond_delta below 1 and the reference group's per-token signal shrinks, so the model is less anchored to the pixels you gave it and the edit gets stronger. Push it above 1 and the deviation is amplified, which locks the output onto the reference. cond_b (upstream's lambda) scales the shared behaviour vector instead - a much blunter instrument.

The teardown numbers are the useful part: measured on 2511 at cfg 4.0 / 24 steps, cond_delta 1.70 gives 0.995 correlation with the input (basically a copy), 1.10 gives 0.997, 0.90 gives 0.928, 0.80 gives 0.711. The cliff sits between 0.9 and 0.8.

The fields worth touching

The defaults are not decoration - cond_b=1.0 with cond_delta=1.0 is a bit-exact no-op, and the node just hands your unmodified MODEL back with no hooks installed. So leave it permanently wired while you tune.

• cond_delta - the dial. Default 1.10, sweep 0.8–1.7 in 0.01 steps. Edit too timid? Go down. Output drifting into a different photograph? Go up.
• img_len - how many trailing image tokens count as the reference group. The 4096 default is exactly a 1024×1024 grid at 16px patches, which is what comfy's TextEncodeQwenImageEdit produces after resizing. Above 1024², or with TextEncodeQwenImageEditPlus and several references, set it to 0 - then the whole image block is reweighted instead of just the tail.
• apply_from / apply_to - a sigma window as denoising progress (0 = pure noise). Defaults cover the whole run. With a 4-step Lightning LoRA the tooltip's own suggestion is 0.0 → 0.6, because late steps are already committed and reweighting there only adds noise.

cond_b, txt_delta and txt_b are 1.0 in every released example. Leave them. verbose prints the effective config, worth one run to confirm it's patching. One output: model, into KSampler.

Installing it

ComfyUI Manager → search "GRAG", or:

cd ComfyUI/custom_nodes
git clone https://github.com/smthemex/ComfyUI_GRAG_Image_Editing

Restart. That's the whole install - zero dependencies (dependencies = [], no requirements.txt), because it's pure comfy-side patching. What it needs is a current ComfyUI: it's a new-style comfy_api extension (comfy_entrypoint, V3 schema), so on an older build the node won't appear in the add-node menu at all. Update first.

Models are whatever your Qwen-Image-Edit workflow already uses - the README wants qwen_image_edit_2511_fp8mixed.safetensors (or a GGUF) in models/diffusion_models, a Qwen-Image VAE, qwen_2.5_vl_7b_fp8_scaled.safetensors in models/clip, and optionally the 4-step Lightning LoRA. And it's a 20B model - quantize it or you won't be running it.

Where people get burned

Don't sweep in 0.1 jumps and conclude it does nothing. Between 1.10 and 1.70 you mostly get variations on "copy".

It is not a "more edit" button. Dropping far below ~0.8 lets the output wander away from the source entirely, not just apply your instruction harder. If you were relying on the reference to keep a face intact, that's the setting that loses it.

The bundled example workflow has a ghost node. example_workflows/grag_2511_native.json uses a GRAGInspect node this release doesn't ship - only GRAGPatch is registered. Comfy flags it as missing; delete it, it was a config read-back helper and not part of the guidance path. The README doesn't mention this.

Stacking attention rewriters is untested ground. The pack chains an override that's already installed rather than clobbering it, which is good engineering - but GRAG plus something like NAG is two rewrites of the same keys in sequence, and nobody has measured that on Qwen Edit.

One honest limit: this buys you control over how much the model listens versus how much it remembers. It won't fix a misunderstood prompt, and it won't rescue an fp8 checkpoint at 4 steps and cfg 1 when the numbers above were taken at cfg 4.0 and 24 steps.