Theory of transverse mode instability in fiber amplifiers with multimode excitations

Transverse Mode Instability (TMI) that results from dynamic nonlinear thermo-optical scattering is the primary limitation to power scaling in high-power fiber lasers and amplifiers. It has been proposed that TMI can be suppressed by exciting multiple modes in a highly multimode fiber. We derive a semi-analytic frequency-domain theory of the threshold for the onset of TMI in narrowband fiber amplifiers under arbitrary multimode input excitation for general fiber geometries. Our detailed model includes the effect of gain saturation, pump depletion, and mode-dependent gain. We show that TMI results from the exponential growth of noise in all the modes at downshifted frequencies due to the thermo-optical coupling. The noise growth rate in each mode is given by the sum of signal powers in various modes weighted by pairwise thermo-optical coupling coefficients. We calculate thermo-optical coupling coefficients for all ∼104 pairs of modes in a standard circular multimode fiber and show that modes with large transverse spatial frequency mismatch are weakly coupled, resulting in a banded coupling matrix. This short-range behavior is due to the diffusive nature of the heat propagation, which mediates the coupling and leads to a lower noise growth rate upon multimode excitation compared to a single mode, resulting in significant TMI suppression. We find that the TMI threshold scales linearly with the number of modes that are excited asymptotically, leading to roughly an order of magnitude increase in the TMI threshold in an 82-mode fiber amplifier.