Adaptive Rectangular and Feature‐Aware Mamba for Nuclei Segmentation in H&E Histopathology Images

Accurate nuclei segmentation in H&E histopathology underpins diagnosis, prognosis, and computational analysis, yet anisotropic shapes, stain/scanner variability, crowded overlaps, and multiscale context make robust delineation difficult. In this paper, we propose Adaptive Rectangular and Feature‐Aware Mamba Network (AFM‐Net), a moderate‐complexity encoder–decoder with skip connections for H&E nuclei segmentation. AFM‐Net integrates two complementary Mamba‐based modules: Adaptive Rectangular Mamba (AR‐Mamba), which estimates height/width‐specific kernel sizes and constructs a dynamic rectangular sampling grid with residual connections for anisotropy‐aware, scale‐adaptive features; and Lightweight Feature‐Aware Mamba (LFAM), which couples a variable‐selection scan with a channel Multilayer Perceptron (MLP) to efficiently process downsampled features while preserving long‐range dependencies in linear time and with a low memory footprint. A hybrid P–Dice loss balances pixel fidelity and region‐level overlap. Extensive experiments demonstrate the superiority of the proposed approach. Results on the publicly available CoNSeP and TNBC datasets show that AFM‐Net outperforms the baseline by 0.122/0.081 in Dice Similarity Coefficient (DSC) and 0.185/0.175 in Aggregated Jaccard Index (AJI), and reduces Hausdorff Distance (HD) by 2.840/2.260, respectively.