Efficient structural information integration into protein language models with implicit mixture-of-experts ProBEiT

Protein language models can now predict function and mutational effects directly from sequence, yet only a few incorporate function-deciding structural information with minimal additional effort. We introduce ProBEiT, a multimodal architecture that jointly encodes amino-acid sequences and discrete, tokenized structures via a parameter-efficient implicit mixture-of-experts scheme with deterministic routing. Trained on 1.7 million AlphaFoldDB proteins, the 400 million-parameter model surpasses larger baselines across six tasks of fitness prediction and functional annotation, demonstrating enhanced understanding of sequence-structure-function relationships. Unlike existing billion-scale models, ProBEiT attains strong accuracy early in continual training period, reducing computational cost. In an experimental thermostability engineering task on glycerol-3-phosphate oxidase, it correctly identifies optimized variants with positive epistatic effects. These results show that tightly coupling structural context with sequence modeling in a modality-specialized framework enables efficient, high-performance protein engineering. ProBEiT integrates protein sequence and tokenized structure with deterministic implicit MoE routing, enabling efficient sequence structure modeling and strong performance in fitness prediction and thermostability engineering.