Antibody language models have emerged as powerful tools in antibody engineering by capturing patterns in sequences to innovate discovery and optimization. While recent advances in machine learning have significantly enhanced this field, current approaches often struggle to capture the full structural complexity inherent in antibody sequences. To address this, we present AbLingua, a family of language models pre-trained on antibody sequences. The largest model in this family contains 1.7 billion parameters and is trained on 1.4 billion sequences, making it the largest encoder-based language model specific to antibodies. AbLingua employs an advanced tokenization method that expands the vocabulary to a level comparable with human language, which enables the model to capture complex structural motifs influencing antibody behavior. Building upon this method, AbLingua introduces an improved pre-training approach that processes amino acid units to better represent structural interdependencies. We demonstrate that AbLingua achieves superior performance across multiple applications, including paratope prediction, neutralizing capacity assessment, and therapeutic antibody design. Furthermore, it excels in the unsupervised classification of B-cell developmental stages and virus-specific antibodies. Our findings demonstrate that the synergy of advanced tokenization, robust scaling laws, and curated datasets constitutes a superior foundation for antibody engineering, significantly driving development efficiency. AbLingua is a large-scale antibody language model that captures structural motifs through novel tokenization. It excels in paratope prediction and therapeutic design, providing powerful tools to enhance efficiency in antibody engineering.
Bai et al. (Mon,) studied this question.