Text to band gap: Pre-trained language models as encoders for semiconductor band gap prediction

We investigate transformer-based language models, including RoBERTa, T5, Llama-3, and MatSciBERT, for predicting the band gaps of semiconductor materials directly from textual descriptions. The inputs encode key material features, such as chemical composition, crystal system, space group, and other structural and electronic properties. Unlike shallow machine learning models, which require extensive feature engineering, or Graph Neural Networks, which rely on graph representations derived from atomic coordinates, pretrained language models can process textual inputs directly, eliminating the need for manual feature preprocessing or structure-based encoding. Material descriptions were constructed in two formats: structured strings with a consistent template and natural language narratives generated via the ChatGPT API. Each model was augmented with a custom regression head and finetuned for band gap prediction task. Language models of different architectures and parameter sizes were all able to predict band gaps from human-readable text with strong accuracy, achieving MAEs in the range of 0.25–0.33 eV, highlighting the success of this approach for scientific regression tasks. MatSciBERT, pretrained on materials science literature, reached comparable performance (MAE 0.288 eV, R 2 0.871) with significantly fewer parameters (110 million), emphasizing the importance of domain-specific pretraining. These results establish that pretrained language models can effectively extract complex feature–property relationships from textual material descriptions and can serve as practical initial property estimators prior to more detailed, structure-based modeling. • Compared general (RoBERTa, T5, LLaMA-3) and domain-specific (MatSciBERT) models. materials property prediction. • Fine-tuned LMs achieve competitive band gap prediction from text inputs. • MatSciBERT shows high efficiency and accuracy via domain pretraining. importance of domain-adaptive pretraining strategies. • Identified scaling laws linking model size, layers, and performance. • Fine-tuning shifts attention toward band gap-relevant features.