Existence and symmetry breaking results for positive solutions of elliptic Hamiltonian systems

Abstract In this paper, we are interested in positive solutions of align*\ array{@{ll} - u = a (x) v^p-1, & in, \\ - v = b (x) u^q-1, & in, \\ u, v>0, & in, \\ u=v=0, & on, array. align* where is a bounded annular domain (not necessarily an annulus) in {R}N (N 3) and a (x), b (x) are positive continuous functions. We show the existence of a positive solution for a range of supercritical values of p and q when the problem enjoys certain mild symmetry and monotonicity conditions. We shall also address the symmetry breaking phenomena where the system is fully symmetric. Indeed, as a consequence of our results, we shall show that problem (1) has N2 (the floor of N2) positive non-radial solutions when a (x) =b (x) =1 and is an annulus with certain assumptions on the radii. In general, for the radial case where the domain is an annulus, we prove the existence of a non-radial solution provided align* (p-1) (q-1) > (1+2NH) ² (qp), align* where H is the best constant for the Hardy inequality on. We remark that the best constant H for the Hardy inequality is just the characteristic of the domain, and is independent of the choices of p and q. For this reason, the aforementioned inequality plays a major role to prove the existence and multiplicity of non-radial solutions when the problem is fully symmetric. Our proofs use a variational formulation on appropriate convex subsets for which the lack of compactness is recovered for the supercritical problem.