From Prototype to Platform: Scalable Machine Learning and Automated Experimentation in Electron Microscopy

Machine Learning (ML) and Artificial Intelligence (AI) have transitioned from a curiosity to a necessity in data-intensive microscopy workflows. Deep learning-based object detection and segmentation frameworks now routinely outperform classical image processing approaches in speed [1], repeatability [2,3], and scalability [4,5]. In parallel with advances in architecture (e.g., transformer-based backbones and lightweight detection heads), substantial progress has been made in addressing a traditional bottleneck of supervised learning: the scarcity of labeled data. The implementation of physics-informed synthetic data generation [6] and generative adversarial networks (GANs) has enabled the creation of realistic microstructural training datasets that incorporate diffraction-contrast physics, defect-morphology constraints, and imaging-noise characteristics. These approaches significantly reduce manual labeling burden while improving model robustness across imaging conditions and microscopes. Despite these advances in algorithms and data, widespread adoption across laboratories has remained limited. The central bottleneck for many is no longer model accuracy or the availability of training data. Instead, the challenge lies in transition: moving from an academic or proof-of-concept ML model to a robust, deployable, cross-laboratory system that integrates seamlessly with microscope hardware, vendor software, and real-time autonomous experimental control. Many ML workflows remain confined to expert users who can manage software stacks, resolve dependency conflicts, and manage offline processing pipelines. The result is that ML often functions as a post-processing tool rather than an integrated component of experimental design and execution. Here, we present the evolution of the Theiascope™ (v11.2+) platform toward closing this transition gap. The newly updated architecture (Fig. 1) integrates and hosts a native Python Jupyter-based development environment, a relational PostgreSQL/SQL data infrastructure, real-time inference pipelines, and direct vendor API communication within a GPU-enabled edge-, near-edge, and/or server-grade computing hardware. This design enables rapid promotion of microscopist-developed notebook scripts into fully deployed, web-accessible, fleet-managed applications in less than one week. All inference outputs, tracking metadata, experimental parameters, and instrument state variables are written to a structured database, enabling real-time querying, retrospective analysis, reproducibility, and cross-laboratory deployment. Vendor API integration further enables bidirectional communication, allowing ML inference to inform active microscope control and autonomous experimental workflows. We demonstrate this transition through two representative studies that originated as conventional scripting-based analyses and were subsequently integrated into the fully deployed platform. First, we address high-temporal-fidelity quantification of dynamic defect evolution in in-situ experiments. A workflow combining Mask R-CNN segmentation, Slicing Aided Hyper Inference (SAHI) for small-object detection in large-format images, and Multi-Object Tracking (MOT) algorithms was initially implemented for offline analysis. Integration into the Theiascope™ platform enables real-time, database-indexed tracking and quantification. The deployed framework is demonstrated on (1) Helium (He) bubble nucleation and growth in He-implanted 316L stainless steel during in situ TEM annealing, representing a thermal extreme, and (2) irradiation-induced dissolution of MX-TiC precipitates in nanostructured alloys imaged in Centered Dark Field (CDF), representing a radiation extreme and alternate imaging mode. Continuous object identity tracking allows automated extraction of growth kinetics, dissolution rates, and interaction statistics across distinct materials systems and imaging conditions (Bright Field versus CDF), illustrating cross-modal robustness of the platform. Second, we demonstrate high-spatial-fidelity analysis via three-dimensional reconstruction of discrete defects using stereoscopic tilts. A YOLO-based Oriented Bounding Box (OBB) detector identifies dislocation loops and cavities, and a MOT framework links features between tilt images. Tilt-displacement vectors are used to reconstruct full 3D spatial coordinates. Validation with synthetic datasets confirms accurate localization and, for crystallographically correlated defects such as dislocation loops, precise determination of habit planes. Application to in situ TEM ion-irradiated 99.99% Ni reveals the complex three-dimensional morphology of the dislocation loop network and strengthens the quantitative correlation between experiments and simulation. Integration into the deployed platform ensures that tilt geometry, tracking vectors, and reconstructed coordinates are stored within the relational database management system (RDBMS) backbone, supporting reproducibility and cross-laboratory scaling. Together, these studies illustrate the transition from isolated ML prototypes to a scalable, automation-enabled microscopy infrastructure. By unifying Jupyter-based development, SAHI-enhanced inference, MOT tracking, relational data management, and vendor API-based experimental control within a deployable GPU-enabled compute framework (Fig. 1), we present a pathway toward widespread cross-laboratory adoption of ML-driven, autonomous experimentation in advanced microstructural analysis and in situ microscopy [7]. Simplified schematic of the revised Theiascope™ (v11.2+) platform showing the integrated rapid prototyping and scripting integration with a full web-stack enabled GPU computing hardware for scalable ML and automated experimentation in electron microscopy.