Revisiting Machine Unlearning with Dimensional Alignment

Machine unlearning, an emerging research topic focusing on data privacy compliance, enables trained models to erase information learned from specific data. While many existing methods indirectly address this issue by intentionally injecting incorrect supervision, they often result in drastic and unpredictable changes to decision boundaries and feature spaces, leading to training instability and undesired side effects. To address this challenge more fundamentally, we first analyze the changes in latent feature spaces between the original and retrained models, and observe that the feature representations of samples not included in training are closely aligned with the feature manifolds of previously seen samples. Building on this insight, we introduce a novel evaluation metric for machine unlearning, coined dimensional alignment, which measures the alignment between the eigenspaces of the forget and retain sets. We incorporate this metric as a regularizer loss to develop a robust and stable unlearning framework, which is further enhanced by a self-distillation loss and an alternating training scheme. Our framework effectively eliminates information from the forget set while preserving knowledge from the retain set. Finally, we identify critical flaws in existing evaluation metrics for machine unlearning and propose new tools that more accurately capture its fundamental objectives.