ComfyUI Node

Dropout

DARE-style pruning for deltas — the n-supermario merge that beats plain averaging

By ljleb·Created 2 years ago·Updated 30 days ago· 101
Dropout
  • deltas (delta)
  • overlap (1.0)
  • overlap_emphasis (0.0)
  • probability (0.9)
  • rescale (1.0)
  • seed (None)
  • recipe
merge_checkpointingfalse

Dropout Mecha Recipe is the pack's implementation of the DARE trick ("drop and rescale"): instead of averaging several deltas together, you randomly drop most of the elements from each delta, sum what survives, and rescale the result back up. The author's own description in the source is "aka n-supermario" - it's the merge that turned "add LoRAs together without them fighting" from a coin-flip into a method.

Why it works: when you average a handful of deltas naively, the deltas mostly cancel each other out and you're left with a weak soup. DARE's insight is that most of a delta's elements are noise anyway - so you prune aggressively, keep the few informative entries per delta, and the merged signal is stronger, not weaker. It's the mechanism behind a lot of the "merge 3+ LoRAs into one" workflows people post.

Inputs that matter

  • deltas (delta) - a MECHA_RECIPE_LIST (default empty) of the deltas you're merging. Build it with Mecha Recipe List; each entry should be a delta (a LoRA loaded via Lora Mecha Recipe, or a Subtract output).
  • probability (0.9) - the drop rate. 0.9 means "keep ~10% of each delta" - the classic DARE setting.
  • rescale (1.0) - how much to scale the surviving sum back up after pruning. 1.0 = full DARE rescale.
  • overlap (1.0) - controls whether deltas can overlap in which elements they keep. Odd values = fully overlapping (independent masks); even values = disjoint regions.
  • overlap_emphasis (0.0) - shifts the overlap behavior; fiddly, rarely touched.
  • seed (None) - reproducibility. Set a value and the same pruning happens every run.
  • merge_checkpointing - the usual branch-cache toggle.

Output: one recipe (MECHA_RECIPE). All of those numbers are themselves recipes, so Float/Int nodes feed them.

The workflow

Lora Mecha Recipe (LoRA 1) ─┐
Lora Mecha Recipe (LoRA 2) ─┼─▶ Recipe List ─▶ Dropout ─▶ Add Difference ─▶ Merger
Lora Mecha Recipe (LoRA 3) ─┘

The pack even ships the standalone pieces (ties_sum_with_dropout), but this node is the convenient composition: drop + rescale the deltas, then add the merged delta back onto the base.

Install & gotchas

ComfyUI Manager → search mechaMecha Merge Node Pack, or:

cd ComfyUI/custom_nodes
git clone https://github.com/ljleb/comfy-mecha.git
pip install -r comfy-mecha/requirements.txt

Restart; only dependency is sd-mecha==1.1.7.

The foot-guns: feeding weight-space models into the deltas socket instead of deltas (the node sums and rescales whatever it's given - if those are full checkpoints you're adding models to models, not deltas). And treating probability like a strength dial: it's a drop rate, so lower = keep more = closer to plain averaging, higher = more aggressive pruning. If you don't set a seed, runs aren't reproducible - which is fine for exploring, annoying when you finally land on a good merge and want the exact same one again.

Categorymecha

Inputs (7)

NameTypeDefaultDescription
deltas (delta)optMECHA_RECIPE_LIST
overlap (1.0)optMECHA_RECIPE1
overlap_emphasis (0.0)optMECHA_RECIPE0
probability (0.9)optMECHA_RECIPE0.9
rescale (1.0)optMECHA_RECIPE1
seed (None)optMECHA_RECIPE
merge_checkpointingoptBOOLEANfalseSpeeds up an entire branch of a merge graph that does not change often in exchange of memory. - true: store the first output of this recipe node on cpu memory in fp16. On subsequent workflow executions, as long as the inputs do not change, the cached keys are returned after being cast to the original device and dtype. - false: do not store the output. The recipe and its inputs will re-execute on subsequent workflow executions. Note that the memory used to checkpoint the output is distinct from the cache feature. In general, you probably want to either use this *or* a cache unit, but not both at the same time because the memory adds up. The difference between merge checkpointing and cache is that merge checkpointing completely re-merges from scratch if any input changes. Merge checkpointing is also generally much faster than cache in the fast path.

Outputs (1)

NameTypeDescription
recipeMECHA_RECIPE