Chiral spin state and nematic ferromagnet in the spin-1 Kitaev- Γ model

The higher-spin Kitaev magnets, in which the Kitaev interaction and off-diagonal exchange couplings are overwhelmingly large, have emerged as a fertile avenue to explore exotic phases and unusual excitations. In this paper, we study the quantum phase diagram of the spin-1 Kitaev-Γ model on the honeycomb lattice using density-matrix renormalization group. It harbors six distinct phases and the intriguing findings are three magnetically ordered phases in which both time-reversal symmetry and lattice symmetry, albeit of different sorts, are broken spontaneously. The chiral spin state originates from the order-by-disorder effect and exhibits an almost saturated scalar spin chirality at the quantum level. Depending on the relative strength of the two interactions, it also features a columnarlike or plaquettelike dimer pattern as a consequence of the translational symmetry breaking. In parallel, the nematic ferromagnets are situated at the ferromagnetic Kitaev side and possess small but finite ferromagnetic ordering. The lattice-rotational symmetry breaking enforces nonequivalent bond energy along one of the three bonds. Although the intrinsic difference between the two nematic ferromagnets remains elusive, the discontinuities in the von Neumann entropy, hexagonal plaquette operator, and Wilson loop operator convincingly suggest that they are separated via a first-order phase transition.