Spiral order by disorder and lattice nematic order in a frustrated Heisenberg antiferromagnet on the honeycomb lattice

Motivated by recent experiments on Bi₃Mn₄O₁₂ (NO₃), we study a frustrated J₁-J₂ Heisenberg model on the two-dimensional (2D) honeycomb lattice. The classical J₁-J₂ Heisenberg model on the 2D honeycomb lattice exhibits N\'eel order for J₂J₁/6, it has a family of degenerate incommensurate spin spiral ground states where the spiral wave vector can point in any direction. Spin wave fluctuations at leading order lift this accidental degeneracy in favor of specific wave vectors, leading to spiral order by disorder. For spin S=1/2, quantum fluctuations are, however, likely to be strong enough to melt the spiral order parameter over a wide range of J₂/J₁. Over a part of this range, we argue that the resulting state is a valence bond solid (VBS) with staggered dimer order---this VBS is a lattice nematic which breaks lattice rotational symmetry. Our arguments are supported by comparing the spin wave energy with the energy of the VBS obtained using a bond operator formalism. Turning to the effect of thermal fluctuations on the spiral ordered state, any nonzero temperature destroys the magnetic order, but the discrete rotational symmetry of the lattice remains broken resulting in a thermal analog of the nematic VBS. We present arguments, supported by classical Monte Carlo simulations, that this nematic transforms into the high temperature paramagnet via a thermal phase transition which is in the universality class of the classical three-state Potts (clock) model in 2D. We discuss the relevance of our results for honeycomb magnets, such as Bi₃M₄O₁₂ (NO₃) (with M=Mn, V, Cr), and bilayer triangular lattice magnets.