Noncollinear Magnetic Order Stabilized by Entangled Spin-Orbital Fluctuations

Quantum phase transitions in the two-dimensional Kugel-Khomskii model on a square lattice are studied using the plaquette mean field theory and the entanglement renormalization Ansatz. When 3z(2)-r(2) orbitals are favored by the crystal field and Hund's exchange is finite, both methods give a noncollinear exotic magnetic order that consists of four sublattices with mutually orthogonal nearest-neighbor and antiferromagnetic second-neighbor spins. We derive an effective frustrated spin model with second- and third-neighbor spin interactions which stabilize this phase and follow from spin-orbital quantum fluctuations involving spin singlets entangled with orbital excitations.