Design of three-mode free-form nanostructured optical fibers: comparison of dense and convolutional neural networks in Generative Inverse Design Networks approach

We report a numerical study on the inverse design of the internal structure of weakly coupled three-mode fibers. We explored a new class of optical fibers – free-form nanostructured fibers (FFNFs) – operating at 1550 nm for potential application in Mode-Division Multiplexing (MDM) systems. The fiber geometries were generated and optimized within the Generative Inverse Design Networks (GIDNs) framework using convolutional neural networks (CNNs) and fully connected dense neural networks (DNNs). The objective of the optimization was to maximize the minimal effective refractive index separation Min|Δ n eff | between supported modes, ensuring weak intermodal coupling. The proposed free-form nanostructured fiber designed with the CNN achieved a minimum modes separation of Min|Δ n eff | = 2.15 × 10 − 3 , exceeding that of a reference three-mode elliptical-core fiber (Min|Δ n eff | = 2.085 × 10 − 3 ). In contrast, the best DNN-optimized structure reached Min|Δ n eff | = 1.99 × 10 − 3 . The results demonstrate that the CNN-based inverse design yields fiber geometries outperforming conventional designs. The proposed methodology can be extended to higher-order mode systems and can include additional fiber properties crucial for telecommunication purposes.