Optimization of biosafety laboratory management via an AI-driven intelligent system

To the Editor: Biosafety laboratories are essential in microbiological research and clinical medicine, as they provide a controlled environment that allows researchers to safely handle and investigate diverse pathogenic microorganisms. The findings from these studies have driven the progress of microbiology and epidemiology, providing critical scientific evidence to address the threats posed by emerging and re-emerging infectious diseases. Nevertheless, working in biosafety laboratories also involves potential risks; in the absence of proper biosafety measures and protocols, incidents of infection or pathogen leakage may transpire. Consequently, it is imperative to provide researchers with systematic training and implement rigorous regulatory mechanisms and emergency response plans. ChatGPT is an artificial intelligence system based on large language models developed by OpenAI, which has attained notable achievements in the domains of natural language processing and generation.[1] ChatGPT has exhibited tremendous potential for applications in the medical field, such as automatically generating medical exam questions and answers, acting as a personalized learning assistant, supporting course design, and aiding in the analysis of medical imaging data.[2,3] ChatGPT is also anticipated to play a pivotal role in the training of biosafety laboratory researchers, offering them an interactive learning experience through intelligent training systems, enabling them to rapidly acquire biosafety knowledge.[4] We investigated whether ChatGPT and other generative AI models could effectively answer questions related to biosafety experiments. First, we collected a dataset containing 62 text-based questions and 8 image-based questions [Supplementary Table 1, https://links.lww.com/CM9/C571], with the text-based questions sourced from biosafety assessment materials of three prominent medical schools (institution-specific policy questions removed) and two open-ended questions from the University of Hong Kong's (HKU) biosafety examination materials. The image-based questions were sourced equally from HKU's biosafety examination materials and the U.S. Centers for Disease Control and Prevention biosafety training modules (https://www.cdc.gov/training/quicklearns/biosafety/). For the text-based questions, we selected Gemini Pro [Figure 1A], Claude-3 [Figure 1B], Claude-2 [Figure 1C], GPT-4 [Figure 1D], and GPT-3.5 [Figure 1E] for evaluation. For the image-based questions, we utilized Gemini Pro Vision [Figure 1F] and GPT-4V [Figure 1G] models for evaluation. We used the same prompt for each question: "As an expert in biology and the administrator of a biosafety laboratory, please provide answers to the following inquiries without any additional explanation". The questions and options were submitted together to the large language models (LLMs). Responses were obtained by calling the application programming interface (API) using the default parameters of each model. To mitigate the impact of LLMs' randomness on model evaluation, each model independently generated three responses for each question. For the calculation of the 95% confidence interval (CI), the z-score was utilized for the 60 text-based questions. For the 8 image-based questions, the t-score was employed. Data analysis was performed using R software (version 4.3.3,https://cran.r-project.org/bin/macosx/big-sur-arm64/base/).Figure 1: Evaluation of LLMs performance on 60 text-based questions and 8 image-based questions related to biosafety laboratories. The performance of Gemini Pro (A), Claude-3 (B), Claude-2 (C), GPT-4 (D), GPT-3.5 (E), Gemini Pro Vision (F), and GPT-4V (G). The right side of the figure displays, from top to bottom, the strict accuracy (the proportion of questions answered correctly all three times), overall accuracy (the proportion of questions answered correctly at least twice), and ideal accuracy (the proportion of questions answered correctly at least once). (H) The bar chart illustrates the comparative performance of LLMs in responding to text-based queries. (I) The bar chart depicts the comparative performance of LLMs in addressing image-based inquiries. LLMs: Large language models; RAAR: Reference answer accuracy rate; SAAR: Subjective answer accuracy rate; SAR: Strict accuracy rate.By comparing the model-generated responses with the standard answers using comprehensive evaluation metrics such as Reference Answer Accuracy Rate (RAAR), Subjective Answer Accuracy Rate (SAAR), and Strict Accuracy Rate (SAR),[5] we analyzed the performance of the models for both text-based and image-based questions. The results demonstrated that LLMs exhibited excellent performance in answering biosafety laboratory-related questions, regardless of whether they were text-based [Figure 1H] or image-based [Figure 1I]. Therefore, we hypothesize that AI models can provide timely, accurate, and comprehensive question-answering services for novice researchers in the field of biosafety. And the detailed definitions of the 3 standardized metrics mentioned above recommended by Lin et al[5] are listed as follows: RAAR= Number of correct answers given by generative AINumber of standard answers SAAR= Number of correct answers given by generative AINumber of answers given by generative AI SAR: 0 (Not totally correct) or 1 (Totally correct) Overall, for text-type questions, Gemini-pro [Figure 1A], Claude-3 [Figure 1B], Claude-2 [Figure 1C], GPT-4 [Figure 1D], and GPT-3.5 [Figure 1E] all demonstrated excellent performance in handling text-based content. Observing the proportion of questions answered correctly in all three questionings, they reached 75%, 74%, 69%, 69%, and 62% respectively. Regarding model performance differences, as shown in Figure 1H, for the 60 text-based questions, Gemini Pro achieves the highest RAAR of 79.4% (95% CI, 69.2%–89.7%], reflecting the AI's accuracy relative to the reference. Claude-3 follows closely at 78.7% (95% CI, 68.3%–89.1%), Claude-2 at 76.5% (95% CI, 65.8%–87.2%), GPT-4 at 75.7% (95% CI, 64.9%–86.6%), and GPT-3.5 at 70.3% (95% CI, 58.8%–81.9%). Compared to the RAAR, all models exhibit a slight decline in the SAAR, which accounts for partial scores when additional options are selected. Gemini Pro still demonstrates the highest SAAR at 77.5% (95% CI, 66.9%–88.1%), followed by Claude-3 at 76.2% (95% CI, 65.4%–87.0%), Claude-2 at 75.2% (95% CI, 64.3%–86.2%), GPT-4 at 73.0% (95% CI, 61.7%–84.2%), and GPT-3.5 at 67.9% (95% CI, 56.0%–79.7%). The SAR requires a 100% match with the reference to score points. Gemini Pro attains the highest SAR at 75.0% (95% CI, 64.0%–86.0%), followed by Claude-3 at 74.4% (95% CI, 63.4%–85.5%), Claude-2 at 73.9% (95% CI, 62.8%–85.0%), GPT-4 at 71.1% (95% CI, 59.6%–82.6%), and GPT-3.5 at 66.1% (95% CI, 54.1%–78.1%). In the assessment of 8 image-based inquiries, GPT-4V exhibited superior performance in comparison to Gemini Pro [Figure 1I]. GPT-4V's RAAR was 78.7% (95% CI, 68.3%–89.1%), with a SAAR at 76.2% (95% CI, 65.4%–87.0%). The SAR for GPT-4V was documented at 74.4% (95% CI, 63.4%–85.5%). Conversely, Gemini Pro exhibited inferior performance. Gemini Pro's RAAR was 76.5% (95% CI, 65.8%–87.2%), with its SAAR at 75.2% (95% CI, 64.3%–86.2%). Gemini Pro's SAR was 73.9% (95% CI, 62.8%–85.0%). This indicates that models still have certain limitations in processing images. Multimodal AI models, such as ChatGPT-4 and Gemini, have demonstrated remarkable capabilities in image understanding and description.[6,7] By deploying these AI-assisted monitoring systems in laboratory settings, comprehensive and real-time surveillance of laboratory operations and experimental practices can be realized. These intelligent systems can automatically detect anomalies and inappropriate behavior, promptly issuing alerts and reminders, thereby significantly enhancing biosafety laboratory management. In practical applications, these AI systems capture critical experiment nodes to generate personalized operation logs and comprehensive sample processing chains, ensuring complete documentation of all laboratory activities. Beyond visual monitoring, these technologies also contribute to predictive maintenance by constructing machine learning models that analyze performance patterns and forecast maintenance requirements for critical safety equipment such as high efficiency particulate air filtration systems. Further enhancing laboratory operations, AI enables the establishment of networked management platforms that centrally integrate disparate control systems, facilitating remote monitoring and operational coordination across complex laboratory environments.[8] Additionally, these AI systems strengthen biosafety training and education by automatically generating customized test questions and learning materials based on relevant regulations, standards, and guidelines, effectively assessing and reinforcing users' understanding while serving as a valuable tool for knowledge dissemination throughout the laboratory community. Generative AI is increasingly influential in biosafety, yet faces notable limitations. Current systems may produce results that are biased, erroneous, or incomplete, while the field lacks sufficient high-quality training data, especially for rare or abnormal events. Real-time data processing for complex biological threats remains inadequate, and network delays can compromise critical operations. Ethical concerns persist, including data privacy, transparency, and automation bias. To address these challenges, it is essential to incorporate uncertainty markers and automated bias detection, while ensuring ongoing human-AI collaborative verification. Standardized data collection and simulated datasets can help alleviate data scarcity for rare scenarios. Ethically, federated learning can protect privacy, explainable AI frameworks offer transparency, and robust human-AI accountability mechanisms ensure safety in decision-making. Funding This study was partly supported by HKU Young Innovator Award 2022 and HKU Seed Funding (No. 2203100624).