PM Breadcrumbs (Mergekit)

Model Breadcrumbs (breadcrumbs, breadcrumbs_ties) Concept: An extension of task arithmetic designed to sparsify task vectors by pruning parameters with both the smallest and the largest absolute magnitudes (often considered outliers). This method operates in two main steps on the task vector (the difference between a fine-tuned model and the base_model): First, a gamma fraction of the parameters with the largest absolute magnitudes are identified for removal. Then, parameters with the smallest absolute magnitudes are identified for removal. The quantity of these smallest parameters to remove is determined such that the final density of parameters retained in the task vector is achieved, after accounting for the largest ones removed. The intention is to isolate and merge the "meaty," mid-range magnitude changes from the task vector, potentially filtering out noise (smallest changes) and overly dominant or conflicting large changes (largest changes). Variants: breadcrumbs: Model Breadcrumbs pruning without TIES sign consensus breadcrumbs_ties: Model Breadcrumbs pruning with TIES sign consensus Use Cases: Merging models where extreme parameter changes might be detrimental or noisy Refining task vectors by focusing on mid-range modifications, removing both the least significant and most extreme changes Inputs: Requires 2 or more models. Key Parameters: weight (per-model): Weight for each model's task vector. gamma (per-model): The fraction of parameters with the largest absolute magnitudes in the task vector to be pruned (removed). For example, a gamma of 0.01 targets the removal of the top 1% of parameters with the highest absolute values. This parameter corresponds to β (beta) as described in the reference paper. density (per-model): The final target fraction of parameters to retain in the task vector after both pruning steps (removal of largest gamma fraction and a corresponding fraction of smallest magnitude parameters). The fraction of parameters with the smallest absolute magnitudes that will be pruned is calculated based on density and gamma. Specifically, it is max(0, 1.0 - density - gamma). Example: If density: 0.9 and gamma: 0.01: The top 0.01 (1%) largest magnitude parameters are removed. The bottom 1.0 - 0.9 - 0.01 = 0.09 (9%) smallest magnitude parameters are also removed. This results in 0.9 (90%) of the parameters being retained. Edge Case: If gamma is set high enough such that gamma >= 1.0 - density (meaning 1.0 - density - gamma <= 0), then the number of largest magnitude parameters actually pruned will be adjusted to 1.0 - density, and no smallest magnitude parameters will be pruned (i.e., the fraction of smallest parameters pruned becomes 0). This ensures the density target is always respected and represents the fraction of parameters kept. lambda (global): As in Task Arithmetic.