Answers are grounded in this paper and cite their sources.
The widespread adoption of the pre-training and fine-tuning paradigm has resulted in a large number of task-specific model checkpoints, motivating increasing interest in model merging as a training-free approach to endow a single model with multi-task capabilities. Existing model merging methods predominantly assume that all models are fine-tuned from the same pretrained checkpoint, an assumption that breaks down when models differ in initialization and exhibit structural heterogeneity. In such settings, direct parameter- or representation-based alignment becomes unreliable, leading to severe performance degradation. In this work, we propose A-EMM, an activation-driven, training-free model merging framework for simple heterogeneous models with different initializations. Instead of relying on weight-space alignment, A-EMM leverages neuron activations to guide the entire merging process through three stages: evaluation, matching, and merging. Specifically, neuron importance is estimated from activation statistics to enable layer-segment matching between heterogeneous models, after which neurons are aligned and merged based on activation similarity within each layer. This activation-centric design allows heterogeneous models to be merged without retraining while preserving their functional behaviors. Extensive experiments on multiple architectures and datasets demonstrate that A-EMM consistently achieves performance comparable to homogeneous model merging and slightly outperforms existing state-of-the-art heterogeneous merging methods, validating its effectiveness and robustness. The code is available at {https://github.com/uestcAs/A-EMM}{https://github.com/uestcAs/A-EMM}.
No takes yet. Share an insight, caveat, or question.
An et al. (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: