BEAT: A Foundation Model for Human Behavioral Sequences

We introduce BEAT (Behavioral Encoder for Action Trajectories), a transformer-based foundation model designed to encode sequences of human behavioral events into dense, reusable embeddings. While large language models have transformed text understanding and vision transformers have advanced image recognition, no equivalent foundation model exists for the modality of human behavioral sequences — the timestamped event trails generated by users interacting with digital products. BEAT addresses this gap with a custom architecture incorporating temporal encoding, event-type embeddings, and three complementary pre-training objectives: masked event prediction, next event prediction, and contrastive learning across temporal windows. Pre-trained on 18,401 behavioral sequences derived from 50,000 users in the REES46 e-commerce dataset (comprising over 20 million raw events), BEAT produces 768-dimensional embeddings that capture complex behavioral patterns without manual feature engineering. We release the model weights, training code, and an interactive demonstration under the Apache 2.0 license to facilitate research in behavioral sequence modeling.