Quantum phase transition between spin liquid and spin nematics in spin-1 Kitaev honeycomb model

Besides the exactly solvable spin-1/2 Kitaev model, higher spin-<a:math xmlns:a="http://www.w3.org/1998/Math/MathML"><a:mi>S</a:mi></a:math> ones, not exactly solvable, are promising playgrounds for researches on the quantum spin liquid as well. One of the main interests in higher spin-<b:math xmlns:b="http://www.w3.org/1998/Math/MathML"><b:mi>S</b:mi></b:math> cases is the interplay between the Kitaev spin liquid (KSL) and spin nematics. We probe this interplay in a spin-1 model on the honeycomb lattice with competing bilinear-biquadratic and Kitaev interactions. Utilizing the 2D infinite projected entangled-pair state (iPEPS), we map out the phase diagram for the ferro-biquadratic interaction. In the phase diagram, we discover the direct phase transitions between the KSL phases and ferro-quadrupolar ordered spin nematic phase, which we call the FQ phase. It has been revealed that the ferro KSL exhibits robustness against perturbations from ferro-quadrupolar interactions. Also, the FQ phase is extended to the parameter region near the antiferro-Kitaev limit. Published by the American Physical Society 2024