An extension for merging LoRAs. Offers a wide range of LoRA merge techniques (including dare) and XY plots. XY plots require efficiency nodes.

LoRA Power-Merger ComfyUI


    Concept: Extends DARE by using adaptive pruning based on parameter magnitudes within each row of the delta 
    parameters (task vectors). 
    It calculates keep probabilities for each parameter: parameters with larger magnitudes within a row 
    are assigned higher probabilities of being kept, while parameters with smaller magnitudes are assigned lower 
    probabilities. 
    These keep probabilities are scaled to range from 
    density - epsilon (for the smallest magnitude element in a row) 
    to 
    density + epsilon (for the largest magnitude element in a row). 
    This method aims to retain important changes while reducing interference, followed by DARE-like rescaling.

    Variants:

    della: DELLA pruning with TIES sign consensus
    della_linear: DELLA pruning without TIES sign consensus
    Use Cases:

    Fine-grained control over pruning by prioritizing parameters with larger magnitude changes
    Combining models where preserving the most significant changes is crucial
    Inputs: Requires 2 or more models, plus one base_model.

    Key Parameters:

    weight (per-model): Weight for each model's task vector
    density (per-model): Target fraction of weights to retain in differences from the base model
    epsilon (per-model): Defines the half-width of the range for keep probabilities. 
    Keep probabilities for parameters in a row will range from density - epsilon to density + epsilon, 
    mapped from the smallest to largest magnitude parameters in that row, respectively. 
    epsilon must be chosen such that density - epsilon > 0 and density + epsilon < 1.
    lambda (global): As in Task Arithmetic
    

ComfyUI Node: PM Della (Mergekit)

Category

Inputs

Outputs

Extension: LoRA Power-Merger ComfyUI

More nodes in LoRA Power-Merger ComfyUI

Other LoRA PowerMerge/Task Arithmetic nodes

Run ComfyUI workflows without the setup