LUNA-FM: a subword foundation model for long non-coding RNAs for functional inference

RNA foundation models (FMs) have shown promise for RNA identification and functional annotation. However, many use single-nucleotide tokenization, which produces long input sequences and can increase computational costs when modelling long RNAs, including long non-coding RNAs (lncRNAs). To address these limitations, we introduce LUNA-FM (Length-efficient sUbword-based Nucleic Acid Foundation Model), an RNA language model pretrained on approximately 28 million redundancy-filtered ncRNA sequences. LUNA-FM combines subword tokenization with a long context transformer, enabling efficient modelling of long RNA transcripts. We systematically evaluate tokenization strategies and vocabulary sizes to investigate the trade-offs between sequence compression, representation granularity, computational efficiency, and downstream performance. Across multiple RNA prediction tasks, including transcript classification, subcellular localization, and RNA-protein interaction prediction, LUNA-FM achieved competitive performance compared with existing RNA FMs while reducing effective input length and fine-tuning cost. We also found that frequency-based subword tokenization generally produced more stable and effective representations than likelihood-based approaches, although the optimal vocabulary size remained task dependent. Training-set motif discovery identified candidate motifs enriched relative to matched low-attribution and dinucleotide-shuffled controls of preeclampsia-associated lncRNAs. Overall, our findings highlight the task-dependent effects of subword tokenization and vocabulary size, extend RNA FM evaluation to longer transcript inputs, and establish LUNA-FM as a competitive model with favourable recorded fine-tuning times. LUNA-FM combines a long-context transformer architecture with subword tokenization for RNA modelling, incorporating alternating attention, improved activation functions, and padding-aware computation to support efficient representation learning across diverse RNA tasks.Across multiple downstream evaluations, LUNA-FM demonstrates competitive performance while maintaining favourable computational efficiency, particularly for long RNA transcripts and lncRNA-related tasks.Systematic tokenization analysis shows that frequency-based BPE tokenization generally outperformed probabilistic Unigram segmentation across the evaluated RNA tasks, including transcript-level classification and splice-site prediction. These results highlight the influence of subword construction strategy on downstream performance.Vocabulary size influences both sequence compression and downstream performance, highlighting a trade-off between preserving local nucleotide information and modelling broader contextual dependencies.Interpretability analysis using Integrated Gradients identified motif-enriched regions associated with model predictions, providing candidates for independent testing of predictive and biological relevance.Subword tokenization reduces the number of tokens required to represent RNA sequences. LUNA-FM achieved competitive predictive performance and generally shorter recorded fine-tuning times under the selected configurations, while some nucleotide-level baseline runs encountered GPU memory constraints. LUNA-FM combines a long-context transformer architecture with subword tokenization for RNA modelling, incorporating alternating attention, improved activation functions, and padding-aware computation to support efficient representation learning across diverse RNA tasks. Across multiple downstream evaluations, LUNA-FM demonstrates competitive performance while maintaining favourable computational efficiency, particularly for long RNA transcripts and lncRNA-related tasks. Systematic tokenization analysis shows that frequency-based BPE tokenization generally outperformed probabilistic Unigram segmentation across the evaluated RNA tasks, including transcript-level classification and splice-site prediction. These results highlight the influence of subword construction strategy on downstream performance. Vocabulary size influences both sequence compression and downstream performance, highlighting a trade-off between preserving local nucleotide information and modelling broader contextual dependencies. Interpretability analysis using Integrated Gradients identified motif-enriched regions associated with model predictions, providing candidates for independent testing of predictive and biological relevance. Subword tokenization reduces the number of tokens required to represent RNA sequences. LUNA-FM achieved competitive predictive performance and generally shorter recorded fine-tuning times under the selected configurations, while some nucleotide-level baseline runs encountered GPU memory constraints.