Flexible unimodal density estimation in hidden Markov models

Hidden Markov models (HMMs) are powerful tools for modelling time‐series data with underlying state structure. However, selecting appropriate parametric forms for the state‐dependent distributions is often challenging and can lead to model misspecification. To address this, P‐spline‐based nonparametric estimation of state‐dependent densities has been proposed. While offering great flexibility, these approaches can result in overly complex densities (e.g. bimodal) that hinder interpretability. We propose a method that builds on shape‐constrained spline theory to enforce unimodality in the estimated state‐dependent densities through enforcing unimodality of the spline coefficients. This constraint strikes a practical balance between model flexibility, interpretability, and parsimony. Through three simulation studies and a real‐world case study using narwhal ( Monodon monoceros ) dive data, we demonstrate the proposed approach yields more stable estimates compared to fully flexible, unconstrained models, thus improving model performance and interpretability. Our method bridges a key methodological gap by providing a parsimonious HMM framework that balances the interpretability of parametric models with the flexibility of nonparametric estimation. This powerful tool allows researchers to derive ecologically meaningful inference from telemetry data while avoiding the pitfalls of overly complex models.