Automating Model Validation for Quantifying System Maturity & Quality Assurance

This paper outlines the program benefits realized by implementing validation-based metrics using automated dynamic queries in Systems Engineering models to bound system maturity, baseline growth, and model deliverable completeness. With Model-Based Systems Engineering (MBSE) models quickly becoming the Systems Engineering dynamic standard for producing and managing deliverables in the Aerospace & Defense Industry, it is important to produce tools/processes that are just as dynamic. To effectively manage the development of systems engineering deliverables for aerospace programs, system models require controls that are both robust and flexible. How we measure that development and progress for MBSE system models remains a nonstandard, until now. Validation-based metrics suites are a query mechanism used to identify system elements that fall into a tailorable criterion, thus increasing quality assurance throughout a model's lifecycle. Programs can create measurable criteria for satisfying Milestone Design Review compliance, tracking deliverable progress/maturity, and even automate requirement text validation against criteria that make it unverifiable. This allows for automated quality control gates that will optimize development and reduce errors. Additional applications for these validation-based metric suites include, but are not limited to, the ability to measure elements for mission criticality and system safety by using dynamic counts that allow users to track/locate specific system information. Validation-based metric suites give all stakeholders and project leadership the assurance of tracking system maturity to measure the progress and changes between baselines. Using these suites, programs can optimize model development, track baseline changes, more easily identify errors before deliveries, and track completeness criteria to give all clients a more complete, validated product. Furthermore, the portability of these suites allows for users to automate validation against models from suppliers/subcontractors prior to integration to increase quality control. By successfully implementing these suites across significant aerospace efforts (concept development, modernization, etc.), it has proven to be both beneficial and successful in cost savings by reducing the amount of rework with up to 38% reductions in review story points achieved. Metrics have provided quantitative insight into agile model development, guiding direct allocation of tasking during program increment and sprint planning. Through the usage of these suites, programs have seen a direct correlation for reducing the time allocated for reviews, reductions in model rework during clean-up activities, and verification of task completeness prior to program increment closeouts. With the application of these validation based metric suites, the opportunities to advance Systems engineering and MBSE increase exponentially to provide a more polished deliverable.