Layer-specific L2 regularisation for efficient forgetting in transformers

Key Points

• The aim is to enhance the efficiency of forgetting information in transformer models using layer-specific L2 regularisation.
• Applied L2 regularisation specifically at different layers of transformers.
• Compared forgetting rates with traditional regularisation methods.
• Utilized benchmarks relevant to neural network performance for validation.
• Layer-specific L2 regularisation significantly improved forgetting efficiency.
• Demonstrated superior performance over standard regularisation techniques.
• Achieved a measurable reduction in information retention across targeted layers.