If You Can’t Use Them, Recycle Them: Optimizing Merging at Scale Mitigates Performance Tradeoffs
1. Abstract
This paper explores a novel approach for model merging. The core idea behind this research is to explore how the
merging of multiple model checkpoints, often discarded during the development of frontier models, can yield a
more effective, Pareto-optimal model that offers a balanced performance across heterogeneous tasks. The authors
utilize an evolutionary optimization algorithm, Covariance Matrix Adaptation Evolution Strategy(CMA-ES) to
determine optimal weights in a linear merge (i.e., weighted sum) of checkpoints from ~100B-parameter models.
They find that even seemingly poor checkpoints meaningfully contribute to improved overall performance across
multiple tasks, outperforming both individual models and existing merging baselines
Figure 1:An overview of our setup. Given models obtained from different LLM training runs, we
optimize linear merging weightings (α1,α2,α3) via iterative search to obtain a model with minimal
task tradeoffs. Each represents a single model, with a to designate its performance on the two tasks. The purple color indicates a Pareto-optimal model,
achieving a good balance between the two tasks without being dominated by other models. We show
tradeoffs between only two tasks since it is easier to visualize.
2. Summary
Context: When training large language models (LLMs), multiple checkpoints emerge, each optimized differently, and many are dropped as “suboptimal.”
Problem: Combining these checkpoints could balance trade-offs across tasks (like instruction following, code generation, reasoning), but traditional approaches focus on expert models, not generalist ones.
Solution: Recycle all checkpoints, even suboptimal ones, through a weighted linear combination. Use an evolutionary strategy (CMA-ES) to find weightings that maximize average performance across diverse benchmarks.
Key Insight: Many checkpoints, even weak-performing ones, can contribute positively. The optimized merged model outperforms single checkpoint baselines and naive merging strategies.
3. Experiments / Methodology
Setup:
Selected ~16 checkpoints (~100B parameters) from the Command R+ training runs which naturally exhibit tradeoffs among diverse model capabilities such as code, math, and instruction following.
Figure 2:Performance of individual models over the seven tasks covering different capabilities. Models 1-8 are the result of supervised finetuning runs, while 8-16 from preference optimization. Held-out tasks (MT-Bench and LBPP) are used to evaluate the resulting merges to make sure the merge optimization process does not overfit to the held-in tasks that we aim to minimize tradeoffs over. MT-Bench rating is scaled by a factor of 10 for better visualization.
Merging Technique:
Merged using linear weight interpolation(akin to model soups), where the merged model is a weighted sum of checkpoint weights.
To ensure these models are representative of different training stages, we select such that:
50% are sourced from the supervised fine-tuning (SFT) training stage, and the other 50% from preference optimization (PO) stage.
The SFT checkpoints involve different training data: some are trained purely on code, some on code and academic data, and some on the general multi-task SFT setup, etc.,
The PO checkpoints include variation of the training objective and training hyperparameters (e.g., warmup ratio).
Optimization Approach:
Evaluation:
1. Benchmarked on multiple tasks: code generation, math (GSM8K), instruction following (IFEval),
multi-task language understanding (MMLUPro), multistep reasoning (MUSR), among others.
2. They reported pass@1 for code tasks i.e., MBPP and LBPP. Used accuracy for GSM8K, MMLU Pro,
and MUSR. For IFEval, we report the prompt-level strict accuracy (Zhou et al., 2023), which measures the percentage of prompts for which all verifiable instructions are followed.
3. Conducted pairwise and three-task trade-off experiments showing merging reduces conflicts better
than uniform or greedy merging
Analysis:
Best merges often include almost all checkpoints (with non-zero weights), underscoring that even “poor” designs contribute.
Fitness improves with more iterations, and recycling more checkpoints yields better outcomes.
Low performing models may lead to optimal merges:Even lower-performing checkpoints can boost aggregated performance, emphasizing the value of diversity among checkpoints.
Recycling benefits from more checkpoint:Including more initial checkpoints obviously extends the search space, which might include a better solution, at the expense of alarger search space.
4. Conclusions
Recycling Checkpoints: Instead of discarding, suboptimal intermediate models can be recycled and jointly leveraged to build better-performing generalist LLMs.
Optimization Works: An evolutionary strategy effectively determines optimal merging weights with no retraining.
Scalability: This method scales to production-scale models (100B parameters) and remains training-free, making it computationally efficient.
Their research shows that it is possible to leverage merging when we have many multi-task trained checkpoints, as opposed to the standard setup of merging experts.
A surprising finding is that even checkpoints which perform relatively bad on subtasks can contribute to an overall better mode
5. Suggestions for Future Work
Here are some thoughtful extensions you could propose:
A. Beyond Linear Merging
Explore non-linear merging strategies (e.g., layer-wise weighted sums, gating mechanisms, or attention-based merge models)
to capture richer interactions between checkpoint parameters.
Explanation:
They suggest trying non-linear merging strategies to capture more complex interactions between parameters across
checkpoints. Non-linear techniques, like spherical linear interpolation (SLERP) or methods that incorporate parameter
importance (e.g., Optimal Brain Iterative Merging (OBIM)), use more complex, non-additive approaches to merge models,
preserving more meaningful information in the high-dimensional weight space and mitigating interference
for potentially better performance. Some examples:
Layer-wise Weighted Sums
Instead of one global weight assign different weights per layer.
Captures the fact that some layers in Model A might be more useful than Model B’s, and vice versa.
Attention-based Merge Models
Use attention (like transformers) to decide how much to "attend" to each checkpoint’s parameters.
B. Multilingual / Multimodal Extensions
Test merging strategies across multilingual or multimodal checkpoints, assessing whether recycling improves robustness across languages and modalities—ties nicely with your interests!
But imagine merging different language models (say, one fine-tuned on French, another on Swahili, another on Japanese).The challenge: languages vary in structure and tokenization. A naive merge might “dilute” language-specific strengths.
The opportunity: recycling strategies could help retain strengths from each language model while boosting generalization across all languages.
Link to Power Point presentation (https://docs.google.com/presentation/d/1cLZKALIc4kjW8KjWGEM64l8BCwyRjVgk/edit?usp=sharing&ouid=109437019500627282995&rtpof=true&sd=true)
Reference:
- https://arxiv.org/abs/2412.04144
.png)
Comments
Post a Comment