Semantic-aware Guidance

If you've fought a scene where the background is burned to a crisp while the character still ignores half your prompt, you've met the problem this node is named after. Classifier-free guidance multiplies the prompt-versus-empty difference by one global scale for the whole image, but a background doesn't need the same steering as the face you actually care about. S-CFG ("Semantic-aware Guidance") measures which pixels each prompt token actually attends to, then rescales CFG per semantic region so everything lands at a uniform guidance level. It's a proper port of the CVPR 2024 paper Rethinking the Spatial Inconsistency in Classifier-Free Diffusion Guidance (arXiv 2404.05384), and it slots into a ComfyUI workflow as exactly one node.

This is a niche, experimental thing, not a daily driver - but if you're on SD 1.5 or SDXL and a scene won't behave, it's worth an afternoon. It only makes sense on U-Net checkpoints with real CFG (SD 1.5, SDXL, Pony, Illustrious, NoobAI - the families that still run CFG 4-7). On distilled models at CFG 1 the whole premise dies: there's no two-pass prediction to rescale, and the node's own math depends on both the conditional and unconditional noise predictions existing.

How it works

Every denoising step, the sampler runs the model twice (once with your prompt, once without) and CFG amplifies their difference. S-CFG hijacks both halves of that.

First it patches the U-Net's self- and cross-attention layers to record attention maps at several resolutions (r2/r4/r8/r16 in the code). From those maps it derives a per-token mask - which pixels each prompt word is looking at - plus a foreground/background split, smoothed with a Gaussian kernel. Then in the sampler's CFG hook it computes the magnitude of the cond − uncond difference inside each region and builds a per-pixel rescale factor:

noise_pred = uncond + guidance_scale * rate * (cond − uncond)

That rate map is clamped between 0.8 and 3.0 (and capped at 15 / cfg), so a background that's getting over-guided gets dialed down while an under-guided subject gets dialed up. Your CFG scale is still in charge; this just makes it work per region instead of per image. The clever part is that it needs no masks from you - the attention maps are the segmentation.

Wiring it in

The input list is one thing: model (MODEL). No strength knob, no scale knob, no region count - it's fully automatic, which is either liberating or infuriating depending on how you like your toys. Wire the checkpoint (after any LoRAs) into model, and the single MODEL output goes into your KSampler's model socket. That's the entire integration. The effect isn't adjustable, so when you compare with and without it, hold the seed fixed - the KB's standing advice about CFG demos applies double here, because a new seed can look like an improvement that isn't one.

Installing

ComfyUI Manager → search "s-cfg" (pack title "Semantic-aware Guidance (S-CFG)"), or:

cd ComfyUI/custom_nodes
git clone https://github.com/shiimizu/ComfyUI-semantic-aware-guidance

Then restart ComfyUI. Good news: no model downloads, no requirements.txt, no heavy dependencies - it's pure Python that leans on ComfyUI's existing patching API, so there's nothing to fetch before it works.

Where people get burned

• It's expensive. Capturing attention maps at multiple resolutions and doing the mask matrix math every step costs real time and VRAM. The README is blunt: large resolutions, like an upscale pass, can OOM. Use it on the base generation, not the hi-res fix.
• One CFG hook per graph. It works by registering sampler_cfg_function, and ComfyUI allows only one node to own that at a time. Other CFG-mangling nodes (AutomaticCFG, DynamicThresholding, that kind of crowd) will fight it - don't stack them in the same chain.
• U-Net or nothing. SD 1.5 through SDXL. Flux and SD3 are transformer-based and have no U-Net cross-attention to harvest, so this silently does nothing useful on them.
• SDXL support is honest guesswork. The original paper never touched SDXL; the author wired it up by counting attention layers and eyeballing which blocks to patch. It works for plenty of people, but treat it as community-grade, not CVPR-guaranteed. And since there's no intensity control, if the effect looks subtle on your checkpoint, that's the node being what it is - not a knob you forgot to turn.