Mechanistic Interpretability of Transformers: Extracting Maximum Values from Lists

The interpretability of artificial intelligence models, particularly machine learning and deep learning models, is a crucial area of research to ensure the safe and reliable deployment of AI systems. This project explores the mechanistic interpretability of transformer models by training a small transformer to perform a synthetic, algorithmic task: finding the maximum value in variable length lists. Inspired by Neel Nanda’s work on mechanistic interpretability, this study aims to reverse engineer the trained transformer model to understand its internal workings. The project involves building a transformer from scratch, training it on the maximum extraction task, and analyzing the model’s attention patterns and decision-making processes. The results provide insights into how transformers solve algorithmic problems, highlighting the differences in approach between models and human reasoning. This research contributes to the broader goal of enhancing the transparency and interpretability of AI models, particularly in understanding their behavior on simple yet fundamental tasks.