Spatiotemporal patterns in a diffusive predator–prey system with nonlocal intraspecific prey competition

Abstract We investigate the effect of nonlocal intraspecific prey competition on the spatiotemporal dynamics of a Holling–Tanner predator–prey model with diffusion. We first establish the criteria for Hopf, Turing, double‐Hopf, and Turing–Hopf bifurcations, and determine the stable and unstable regions of the positive equilibrium. For Turing–Hopf bifurcation, by analyzing the normal form truncated to the third order, we derive that, with strong nonlocal interaction, the system exhibits the tristable phenomena, that is, the coexistence of a stable spatially nonhomogeneous periodic orbit and two nonconstant stable steady states, as well as the existence of periodic orbits with two spatial wave frequencies induced by the nonlocal interaction. The main analytical difficulty arises from the nonlocal interaction that prevents the direct application of formulas for the coefficients of the normal form. Biologically, the emerging spatiotemporal patterns suggest that the global intraspecific competition can promote the coexistence of the prey and predator by allowing the prey maintain a critical total population size, which may provide an alternative approach in explaining the group formation of some prey species under the risk of predation. Some coexistence patterns are destabilized by introducing another prey species with local intraspecific competition, leading to the coexistence of two preys and one predator, namely, the prey with nonlocal interaction is concentrated at a single spatial location, and the other prey is distributed uniformly in the rest of the habitat. Accordingly, the predator is forced to change its behavior as well.