High-efficiency terahertz third harmonic generation from quasi-bound states in the continuum in permittivity-asymmetric hybrid metasurface

Optical resonators with high quality (Q) factors are highly desirable in meta-optic and nanophotonic fields. Recently, bound states in the continuum (BIC) have emerged as a pivotal solution to enhance the Q factors of optical resonators to theoretically infinite values, leveraging non-dissipative energy confinement, specific resonance conditions, and destructive interference. However, in all-dielectric structures, the field distribution of BIC is predominantly confined within the metasurface (MS), posing significant challenges to the precise control of mode volume. Here, we propose a metal-dielectric hybrid MS that supports two symmetry-protected BIC, both of which are topologically protected by polarization vortices. Notably, the BIC mode volume is drastically reduced via coupling with surface plasmon polaritons, forming a hybrid mode with enhanced field localization. Specifically, we achieve high-efficiency terahertz third harmonic generation (THG) by breaking the dielectric constant symmetry of the structure to excite quasi-BIC. When the incident power of the fundamental frequency wave reaches 1 kW/cm2, the THG conversion efficiency peaks at an impressive 1.5%. Our findings not only provide a pathway for designing optical components with both high Q factors and low mode volumes but also offer a feasible strategy to boost nonlinear optical effect, opening new avenues for advanced nanophotonic applications.