Habitat complexity alters the strength of sexual selection on female brain size in a livebearing fish.

Animals often reproduce in complex environments, which should generate selection for both enhanced detectability in signaling traits and improved cognitive processing abilities. However, the extent to which signaling and cognitive traits have evolved to overcome the challenges of interacting in complex habitats remains understudied. We examined whether habitat complexity influences sexual selection in the pygmy halfbeak, Dermogenys collettei, a small livebearing freshwater fish. Using free-swimming arenas, we created low- and high-complexity environments and observed mating behaviors in mixed-sex groups. While the opportunity for sexual selection did not differ significantly between environments for either sex, we observed positive selection gradients for female brain size in open arenas, but not in complex habitats. Selection on morphological traits associated with visual signaling was also primarily detected in open environments, particularly in females. These results suggest that habitat complexity may reduce selection pressures on both cognitive traits, such as brain size, and signaling traits relevant to mate choice. Together, our findings highlight the importance of integrating cognitive traits into sexual selection theory and considering sex-specific selection across ecologically relevant contexts.