Stability of bosonic atomic and molecular condensates near a Feshbach resonance

We explore the Bose condensed phases of an atomic gas on the molecular side of a Feshbach resonance. In the absence of atom-molecule and molecule-molecule scattering, we show that the atomic condensate is either a saddle point of the free energy with energy always exceeding that of the molecular condensate, or has a negative compressibility (hence unstable to density fluctuations). Therefore no phase transition occurs between the atomic and molecular condensates. We also show that a repulsive molecule-molecule scattering can stabilize a sufficiently dense atomic condensate, leading to the possibility of a phase transition. We caution that three-body processes may render this transition unobservable.