Apply Visual Style Prompting

If you want a generation to pick up the look of a reference image - the colors, the shading, the texture - without training a LoRA or downloading a single adapter file, this is the node. It's the ComfyUI port of Naver AI's Visual Style Prompting paper from March 2024, and the trick it uses is genuinely different from the IP-Adapter / reference-ControlNet crowd it hung around with: it rewires the self-attention inside the UNet at inference time. No new weights, no model downloads, nothing to tune except the prompt.

There's a catch, and it's in the first sentence of the README: SD 1.5 only. The author has only tested it with 1.5-based models, and there it works because the method digs into the CrossAttention layers of the UNet architecture - the exact anatomy that SDXL and Flux no longer share. If you're on a newer base, this won't help you; IP-Adapter or an edit model will. This node is the 2024 play that the ecosystem left behind, in the same way reference-only ControlNet and the other self-attention-injection tricks got displaced. For a 1.5 workflow it's still a neat, free trick.

How it actually works

Every denoise step, the node builds a two-image latent batch: slot 0 is your VAE-encoded reference, slot 1 is the image being generated. Then it patches the UNet's cross-attention layers with a VisualStyleProcessor that replaces their forward pass. The processor computes keys and values as usual, but then calls swapping_attention, which takes the keys and values from the reference slot and broadcasts them to the generation slot. Your output is literally attending to the reference's features at every layer - that's the whole mechanism. There's also an adain function in the utils that does a mean/std style-normalization pass, the standard style-transfer lever.

The two-slot batch is why you always get two images out. It uses the same VAE-encode-plus-noise-mask machinery as inpaint workflows: the reference slot gets a noise mask of 0 (kept as-is), the output slot gets a mask of denoise, and the pair denoise side-by-side each step. Slice off the reference with a Batch Index Select node and feed index 1 to the decoder.

The inputs that matter

Most are wiring, not knobs. You feed in model and clip from your checkpoint, reference_latent from a VAE Encode of your reference image, and reference_cond from a CLIP Text Encode describing that reference - then your normal positive and negative. The node concatenates reference_cond into the positive conditioning internally, so the model knows what it's looking at.

The actual dials:

• enabled - true by default. Beware the README's warning: flipping it off does not unhook the patched attention. The node stays wired into the model until you remove it.
• denoise (0–1) - how much the output slot is regenerated. 1.0 means full txt2img-style denoise. The author describes this as experimental, and fair warning: the optional init_image input is declared in the code but never actually used by the shipped version, so don't build an img2img pipeline around it expecting real results.
• input_blocks, middle_block, output_blocks + their skip_*_layers - which encoder, middle, and decoder layers get the attention swap, and how many layers from each are skipped. Defaults: input and middle off, output on with 24 of 48-ish layers skipped. That's a sane starting point; crank the toggles if the style isn't coming through or it's coming through too hard.

Outputs: model, positive, negative, and latents (the two-slot batch with its noise mask). Wire the model straight into your sampler.

Install and gotchas

The best part of this pack is that it installs clean - there's no requirements.txt and no model files to fetch, because it only needs torch and einops, both of which ComfyUI already ships. Via Manager, search "Visual Style Prompting" (the pack title is ComfyUI_VisualStylePrompting), or:

cd ComfyUI/custom_nodes
git clone https://github.com/ExponentialML/ComfyUI_VisualStylePrompting

Restart ComfyUI and the node appears as Apply Visual Style Prompting under the VisualStylePrompting/apply category.

Where people get burned: it's marked WIP, and a major rewrite changed how it works - old saved workflows will break, so grab the updated example from the workflows folder. And remember the two-outputs gotcha; everyone's first run has a reference image sitting in their output folder. Finally, the author's one tip is the one that actually matters: engineer both the positive and reference_cond prompts. The example uses orange fox, origami, deep colors, shading, canon 60d for the output and origami figurine for the reference - put what you want to derive in the reference cond, what you want to see in the positive.