Answers are grounded in this paper and cite their sources.
Decoder-only language models have advanced rapidly through architectural innovations such as new normalization methods, gated activations, and efficient attention mechanisms. However, encoder-only architectures, which are excellent at capturing contextual information for tasks like sequence classification and Natural Language Inference (NLI), have largely retained the original BERT family’s design principles. This gap creates particular challenges for low-resource languages like Turkish, where the scarcity of domain-specific pretrained encoders limits progress in specialized areas such as clinical natural language processing (NLP). Although English biomedical and clinical encoders have demonstrated that domain-adaptive pretraining improves biomedical and clinical NLP performance, comparable resources for Turkish remain underdeveloped. To address this, we present Atlas bidirectional encoder representations from transformers (BERT), a modernized BERT-style encoder that integrates recent innovations from both encoder and decoder models for Turkish general and clinical language inference. Inspired by models like ModernBERT, NeoBERT, and EuroBERT, our architecture employs pre-RMSNorm, SwiGLU feed-forward layers, and PyTorch FlexAttention with document-aware masking to extend the context window to 2,048 tokens while preserving document boundaries. Additionally, to address Turkish’s complex agglutinative morphology, we employ a SentencePiece Unigram tokenizer for subword segmentation. AtlasBERT is trained in two stages, first on 35.68 billion Turkish tokens from FineWeb-2, then with continued domain-adaptive pretraining on 791 million tokens of Turkish biomedical text. AtlasBERT achieves 87.82% mean test accuracy on Turkish Stanford natural language inference (SNLI), compared with BERTurk’s 87.03%, while in the clinical domain BioAtlasBERT reaches 77.80% on TurkMedNLI against a BioBERTurk baseline of 75.28%. These results show that combining modern encoder components with Turkish-tailored tokenization and two-stage pretraining provides a competitive foundation for Turkish general and clinical NLI.
No takes yet. Share an insight, caveat, or question.
Oğul et al. (2026) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: