Exploring ultralight dark matter self-coupling via the gravitational wave background

Supermassive black hole (SMBH) binary mergers are a primary source of the stochastic gravitational wave background (SGWB) in the nanohertz band, now being actively probed by pulsar timing arrays (PTAs). We investigate how ultralight dark matter (ULDM) with a quartic selfinteraction, forming solitonic cores around SMBHs, modifies the SGWB through dynamical friction on the inspiralling binary. Solving the Gross-Pitaevskii-Poisson (GPP) equations numerically for halo masses M halo ∼ 10 13 M⊙, we compute self-consistent solitonic density profiles across a range of dimensionless self-coupling parameters λ, covering both attractive and repulsive self-interactions. We find that soliton-induced dynamical friction imprints a characteristic suppression feature in the GW strain spectrum, whose frequency location and depth are sensitive to both the ULDM mass m and the coupling λ. For SMBH masses M• ∼ 10 8.5 M⊙ and ULDM masses in the range m ∼ (3 × 10 -22 -2 × 10 -21 ) eV, current PTA observations from NANOGrav, EPTA, and PPTA may constrain the attractive and repulsive self-coupling of ULDM particles. ULDM masses exceeding ∼ 1.6 × 10 -21 eV are independently excluded by soliton accretion-timescale constraints. These results demonstrate that the nanohertz GW spectrum provides a complementary probe of ULDM self-interactions, operating in a regime distinct from conventional particle-scattering or rotationcurve analyses.