Sign-changing Solutions and Phase Separation for an Elliptic System with Critical Exponent

We study the following elliptic system with critical exponent: Here, Ω is a smooth bounded domain of ℝ N (N ≥ 6), is the critical Sobolev exponent, 0 < λ1, λ2 < λ1(Ω) and μ1, μ2 > 0, where λ1(Ω) is the first eigenvalue of − Δ with the Dirichlet boundary condition. When β = 0, this turn to be the well-known Brézis-Nirenberg problem. We show that, for each fixed β <0, this system has a sign-changing solution in the following sense: one component changes sign and has exactly two nodal domains, while the other one is positive. We also study the asymptotic behavior of these solutions as β → − ∞ and phase separation appears. Precisely, two components of these solutions tend to repel each other and converge to solutions of the Brézis-Nirenberg problem in segregated regions.