Chiral Spin Density Wave Order on the Frustrated Honeycomb and Bilayer Triangle Lattice Hubbard Model at Half-Filling

We study the Hubbard model on the frustrated honeycomb lattice with nearest-neighbor hopping t₁ and second nearest-neighbor hopping t₂, which is isomorphic to the bilayer triangle lattice, using the SU(2)-invariant slave boson theory. We show that the Coulomb interaction U induces antiferromagnetic (AF) chiral spin density wave (χSDW) order in a wide range of κ=t₂/t₁ where both the two-sublattice AF order at small κ and the decoupled three-sublattice 120^∘{} order at large κ are strongly frustrated, leading to three distinct phases with different anomalous Hall responses. We find a continuous transition from a χSDW semimetal with the anomalous Hall effect to a topological chiral Chern insulator exhibiting the quantum anomalous Hall effect, followed by a discontinuous transition to a χSDW insulator with a zero total Chern number but an anomalous ac Hall effect. The χSDW is likely a generic phase of strongly correlated and highly frustrated hexagonal lattice electrons.