Photonic spin Hall effect induced by collective strong coupling of molecular ensembles in a plasmonic nanocavity

We propose a molecular cavity optomechanical (MCOM) framework that exploits the collective vibrational modes of a molecular ensemble coupled to a plasmonic nanocavity. Our approach replaces traditional mechanical elements in cavity optomechanics with molecular vibrations, allowing optomechanically induced transparency (OMIT) to emerge even in regimes with ultralow optical quality factor (Q) that are typically inaccessible to standard cavity optomechanics. We observe that the OMIT window exhibits gain at very low Q factor and with a large number of molecular ensembles. Specifically, we demonstrate dynamic control of the photonic spin Hall effect (PSHE) by tuning the optical susceptibility through parameters such as molecular resonance, collective coupling strength, and the Q factor. Our method first computes the coefficients of the transverse electric (TE) and transverse magnetic (TM) components of reflected light using the Fresnel formula, demonstrating the transverse shift of reflected light from positive to negative at the Brewster angle. Our findings suggest that molecular ensembles are a viable platform for room-temperature nanophotonics, facilitating spin-controlled signal processing and quantum-enhanced sensing through integrated optomechanics and spin-orbit coupling.