PM Della (Mergekit)

Concept: Extends DARE by using adaptive pruning based on parameter magnitudes within each row of the delta parameters (task vectors). It calculates keep probabilities for each parameter: parameters with larger magnitudes within a row are assigned higher probabilities of being kept, while parameters with smaller magnitudes are assigned lower probabilities. These keep probabilities are scaled to range from density - epsilon (for the smallest magnitude element in a row) to density + epsilon (for the largest magnitude element in a row). This method aims to retain important changes while reducing interference, followed by DARE-like rescaling. Variants: della: DELLA pruning with TIES sign consensus della_linear: DELLA pruning without TIES sign consensus Use Cases: Fine-grained control over pruning by prioritizing parameters with larger magnitude changes Combining models where preserving the most significant changes is crucial Inputs: Requires 2 or more models, plus one base_model. Key Parameters: weight (per-model): Weight for each model's task vector density (per-model): Target fraction of weights to retain in differences from the base model epsilon (per-model): Defines the half-width of the range for keep probabilities. Keep probabilities for parameters in a row will range from density - epsilon to density + epsilon, mapped from the smallest to largest magnitude parameters in that row, respectively. epsilon must be chosen such that density - epsilon > 0 and density + epsilon < 1. lambda (global): As in Task Arithmetic