Accelerating Realization of Effective Capacity in Lightweight Vision Models via Self-Competitive Distillation

We introduce Self-Competitive Distillation (SCD), a parameter-neutral training strategy aimed at influencing optimization dynamics without increasing model size or relying on external teachers. Two identical instances of the same architecture, initialized with different random seeds, are trained jointly and dynamically exchange asymmetric teacher–student roles based on instantaneous performance, enabling knowledge transfer between diverging optimization trajectories. Under fixed parameter and training budgets, SCD is observed to improve the realized effective capacity of lightweight architectures, yielding a higher test accuracy at matched epochs. Across multiple lightweight vision models and datasets, SCD demonstrates gains in both in-domain performance and cross-domain generalization, as measured by xScore. These results suggest that, within the evaluated experimental conditions, SCD can help mobile models make more effective use of training dynamics, while the underlying architecture remains the primary determinant of effective capacity in resource-constrained settings.