DARE Merge LoRA Stack

Stacking LoRAs is the standard trick - style + character + concept at once - but the more you pile on, the more they fight each other. Additive loading smears every LoRA's delta over every other, and you end up with the "LoRA stew" look where nothing quite lands. DARE Merge LoRA Stack is an alternative way to combine them, from the DARE paper ("Language Models are Super Mario", arXiv 2311.03099): instead of summing everything, randomly drop parts of each LoRA and rescale what's left.

It's the centerpiece of an experiment by ntc-ai, the SDXL LoRA maker who ships a lot of small, single-concept "slider" LoRAs - exactly the kind of thing this technique shines on, since each one is a focused nudge rather than a whole style baked in.

How it works

For each LoRA in the stack, the node treats the weights as a delta over the base model. It applies DARE per tensor: a Bernoulli mask drops each weight with probability p, then the survivors get rescaled by 1/(1-p) so the expected value of the delta survives the culling. Each dropped-and-rescaled delta is scaled by lambda_val (and the per-LoRA strength from the stack), and all of them are summed into one merged LoRA. The result is a single LoRA-typed value you can apply or save - it's the bridge between a LORA_STACK (what stacker nodes produce) and the LoRA type that Apply LoRA and Save LoRA consume.

Then there's scale, the "experimental lipschitz regularizer" from the README. If scale is positive, the merged weights get normalized by their largest spectral norm so nothing blows past the cap. Set it to -1 (the README's instruction) - any non-positive value works - to turn it off.

The inputs that matter

• lora_stack - the list of LoRAs, from a LoRA Stacker node that outputs LORA_STACK (the demo workflow ships one).
• p - dropout probability, default 0.13. Higher = more aggressive culling. 0.01–1.0.
• lambda_val - multiplier applied to each LoRA's contribution, default 1.5. This is your overall strength dial.
• scale - spectral-norm cap, default 0.2. Leave it positive until you know what you're doing.
• seed - the Bernoulli mask is seeded by torch.manual_seed(seed), so the same stack + settings + seed reproduces the same merge. Different seed, different weights get dropped.

Output is a single LoRA, wired into Apply LoRA (to use it) and/or Save LoRA (to keep it). Worth noting this node is filed under Comfyroll/IO in the node menu, not under the model-merging category with its pack-mates - a surprise if you're hunting for it.

A real gotcha: nothing divides by the number of LoRAs. The merged delta is a raw sum, so each extra LoRA pushes the magnitude up - which is exactly what scale's normalization is there to contain. Turn the regularizer off with many LoRAs and you can overshoot into broken output fast.

Install

Install the pack once for all three nodes:

• ComfyUI Manager: search "ComfyUI - Apply LoRA Stacker with DARE" and install.
• Manually:

  cd ComfyUI/custom_nodes
  git clone https://github.com/ntc-ai/ComfyUI-DARE-LoRA-Merge

  Restart ComfyUI. No requirements.txt and no pip installs - torch, safetensors and PIL already ship with ComfyUI, and no model files are downloaded by the nodes themselves.

Troubleshooting

If you feed it a LORA_STACK and get nothing out, the stacker isn't producing the right type - the node expects LORA_STACK, and a mismatched stacker silently breaks the chain. Reproducibility complaints usually trace to the seed: same p and lambda_val but a new seed means new dropout, and if you're comparing runs you'll chase ghosts. And while the README's before/after images look great with the author's own slider LoRAs, this is a research experiment, not a guaranteed win - the community has reported merges that do nothing on some models. Try it on your own stack, but don't rip out your trusty additive workflow for it until you've seen the difference with your own eyes.