A hybrid mode intensity multiplexing method for OAM communications with crosstalk-bounded waists

Orbital angular momentum (OAM)-based multiplexing enables high-capacity optical communication. However, conventional fixed-waist OAM multiplexing systems have limited channel constellations due to beam divergence and finite-aperture truncation. We propose a hybrid mode intensity multiplexing (MIM) framework that jointly varies beam waist, amplitude, and wavelength for efficient high-capacity OAM communications. Beam waists are selected to meet a pre-calibrated crosstalk bound using spacing rules that account for the Gouy phase, and the OAM states are combined with wavelength-division multiplexing to enhance the capacity. We demonstrate that a well-conditioned channel matrix with pre-calibrated beam waists shows stable capacity scaling. This results in a threefold increase in aggregate spectral efficiency and a 2.5-fold improvement in capacity over (i) fixed-waist OAM multiplexing systems and (ii) an OAM-MIMO spatial multiplexing baseline under identical aperture, power, and misalignment settings.