Rethinking free-space orbital angular momentum quantum key distribution: encoding selection and branch-cut-length-minimized adaptive optics

High-dimensional orbital-angular-momentum quantum key distribution (OAM-QKD) in free space is jointly limited by aperture-induced diffraction and turbulence-induced modal crosstalk. We develop a numerical framework that first selects an encoding for finite-aperture propagation and then addresses the breakdown of conventional adaptive optics (AO) in branch-point-rich turbulence. Comparing Laguerre–Gaussian, perfect Laguerre–Gaussian, and minimum-diffraction-loss (MDL) modes identifies MDL modes as the most robust of the tested encodings against diffraction-induced quantum dit error rate (QDER). Using this baseline, we introduce branch-cut-length-minimized AO: branch points are localized from beacon-wavefront slopes, equivalent hidden-phase representations are ranked by intensity-weighted branch-cut length, and the selected reconstruction is fitted to a finite-actuator deformable mirror. Under moderate-to-strong turbulence, conventional AO provides useful recovery only while the wavefront remains predominantly smooth, whereas branch-cut-length minimization reduces QDER once branch points become prevalent. For a five-dimensional MDL-encoded QKD link at r 0 = 0.01 m, the mean QDER over 500 turbulence realizations decreases from 0.5550 for traditional AO to 0.5302 after branch-cut-length minimization, compared with 0.4957 for the Ideal AO reference. Residual-error analysis further shows that the remaining QDER is most strongly associated with residual phase mismatch and out-of-subspace leakage. At the same time, receiver-plane amplitude scintillation remains inaccessible to phase-only AO. These results establish branch-cut-length minimization as a physically implementable route to extend beacon-based AO beyond smooth-wavefront conditions, while defining the residual limits imposed by the finite measurement subspace and amplitude distortion.