Ambiguity in LLM-Based UML Generation: A Systematic Literature Review of Errors, Mitigation, and Validation

# Protocol and Replication Package for: Ambiguity in LLM-Based UML Generation: A Systematic Literature Review of Errors, Mitigation, and Validation This deposit contains the review protocol, the search strategies, the screening logs, the extracted dataset, the quality assessment scores, and the analysis scripts supporting the systematic literature review (SLR) reported in the manuscript: > Rami, D., El Bahri, A., Zougari, O., Kharmoum, N., & Ziti, S. *Ambiguity in LLM-Based UML Generation: A Systematic Literature Review of Errors, Mitigation, and Validation.* (2026). Manuscript under review. ## Review scope The review synthesizes 62 primary studies published between January 2023 and December 2025 on the use of large language models (LLMs) for transforming natural-language software requirements into Unified Modeling Language (UML) models. It addresses ten primary research questions (RQ1–RQ10) and four secondary review questions (SRQ1–SRQ4), covering: - requirement types transformed into UML models;- UML diagram types targeted;- LLM families, versions, and adaptation strategies;- validation mechanisms (syntactic, structural, semantic, traceability);- datasets and benchmarks;- evaluation metrics;- error and hallucination patterns;- reproducibility artifacts;- research gaps and a research agenda. ## Methodology The review follows the PRISMA 2020 statement (Page et al., 2021) and the Kitchenham & Charters (2007) guidelines for software engineering systematic reviews. Primary studies were retrieved from Scopus, Web of Science, IEEE Xplore, and the ACM Digital Library, using the database-specific queries documented in `search_strings/`. The full review protocol is provided in `protocol.pdf`. ## Contents of this deposit | File / folder | Description ||---|---|| `protocol.pdf` | Full review protocol: research questions, eligibility criteria (IC1–IC6, EC1–EC6), search strategy, screening procedure, data extraction form, quality assessment checklist (QA1–QA10), synthesis plan. || `search_strings/` | One text file per database (`scopus.txt`, `wos.txt`, `ieee.txt`, `acm_abstract.txt`, `acm_title.txt`) with the exact query, execution date, filters, and number of records retrieved. || `screening_logs/` | Title/abstract and full-text screening decisions with study identifiers and exclusion reasons (EC1–EC6). || `extraction_dataset.xlsx` | The 62 included primary studies coded across all fields of the extraction form (identification, requirement type, UML diagram type, LLM family/version, adaptation strategy, validation layer, metrics, error categories, reproducibility artifacts). || `quality_assessment.xlsx` | QA1–QA10 scores per included study, with total score and quality band (High / Moderate / Low). || `scripts/` | Python scripts used for deduplication (DOI, normalized title, fuzzy Jaccard matching ≥ 0.90), automatic keyword pre-screening, descriptive statistics, and figure generation. || `README.md` | This file. | ## Software environment - Python 3.11- pandas, matplotlib, rapidfuzz, openpyxl- Deduplication threshold: Jaccard similarity ≥ 0.90 on normalized titles- The scripts are provided as-is and were run on the dataset included in this deposit. ## How to cite If you use the protocol or the extraction dataset, please cite both the manuscript and this deposit: > Rami, D., El Bahri, A., Zougari, O., Kharmoum, N., & Ziti, S. (2026). *Protocol and replication package for: Ambiguity in LLM-Based UML Generation: A Systematic Literature Review*. Zenodo. https://doi.org/10.5281/zenodo.XXXXXXX ## License The protocol, dataset, and scripts are released under the Creative Commons Attribution 4.0 International license (CC BY 4.0). You are free to share and adapt the material for any purpose, provided that appropriate credit is given. ## Contact Driss Rami — drami.cpr@gmail.com — Intelligent Processing and Security of Systems (IPSS) Laboratory, Faculty of Sciences, Mohammed V University, Rabat, Morocco.