Nonlocality-induced modulational instability in coupled Hartree equations

The emergence of standing-wave states and modulational instability in a system of coupled nonlocal nonlinear Hartree equations is considered. Such equations naturally arise in the mean-field description of two-component Bose-Einstein condensates and in nonlinear optical media with long-range interactions. By analytical stability analysis, applying a Lens-type transformation, we determine the modulational instability gain spectrum and identify the conditions under which uniform (steady) states become unstable. We consider a quadratic external potential of the form Vj(x) = λjx2, where the parameter λj may take positive, negative, or zero values, allowing us to explore trapping, expulsive, and homogeneous configurations for all types of atomic interactions. Numerical simulations reveal rich dynamical behavior characterized by the onset of phase segregation and overlapping patterns in the unstable regime. The interplay between strong nonlinearity and nonlocal coupling drives spontaneous structure formation and collective excitation modes in coupled nonlocal systems.