Investigating Novice Modelers’ Intuitive Consistency Notions for the Case of Compositional Models

The terms “consistency” and its counterpart, “inconsistency”, are omnipresent in model-driven software engineering. Particularly during model evolution, consistency assurance is a central concern. Technically, consistency can be defined from various viewpoints. However, it is unclear how these definitions of consistency align with practitioners’ mental models. Understanding how practitioners assess and respond to different kinds of consistency problems in their own minds is required for realizing successful consistency assurance. An important aspect towards this goal is the investigation of whether there is a shared mental model of consistency among an evenly experienced group of practitioners. The current state of the art gives no answer to this. In this work, we present a model dataset of 62 consistency problems based on simple, compositional models. For this dataset, we collected 452 consistency ratings from a student group and one rating per problem from an expert group. The results show that there is no common understanding, i.e., there is no shared mental model of consistency within a group or between groups. Still, common patterns in the ratings are observable, such as a preference for rating consistency on a fine-grained scale rather than as a boolean property, or that syntactic contradictions are much less relevant to ratings than subjective semantic interpretations. All developed tools, models, and collected data are openly available for future research.