Closing the Loop: Evaluating a Measurement Instrument for Maturity Model Design

To support the systematic improvement of business intelligence (BI) in organizations, we have designed and refined a BI maturity model (BIMM) and a respective measurement instrument (MI) in prior research. In this study, we devise an evaluation strategy, and evaluate the validity of the designed measurement artifact. Through cluster analysis of maturity assessments of 92 organizations, we identify characteristic BI maturity scenarios and representative cases for the relevant scenarios. For evaluating the designed instrument, we compare its results with insights obtained from in-depth interviews in the respective companies. A close match between our model's quantitative maturity assessments and the maturity levels from the qualitative analyses indicates that the MI correctly assesses BI maturity. The applied evaluation approach has the potential to be re-used in other design research studies where the validity of utility claims is often hard to prove.