Prototyping a process monitoring experiment

Features are often the basic unit of development for a very large software system and represent long-term efforts, spanning up to several years from inception to actual use. Developing an experiment to monitor (by means of sampling) such lengthy processes requires a great deal of care in order to minimize casts and to maximize benefits. Just as prototyping is often a necessary auxiliary step in a large-scale, long-term development effort, so, too, is prototyping a necessary step in the development of a large-scale, long-term process monitoring experiment. Therefore, we have prototyped our experiment using a representative process and reconstructed data from a large and rich feature development. This approach has yielded three interesting sets of results. First, we reconstructed a 30-month time diary for the lead engineer of a feature composed of both hardware and software. These data represent the daily state (where the lead engineer spent the majority of his time) for a complete cycle of the development process. Second, we found that we needed to modify our experimental design. Our initial set of states did not represent the data as well as we had hoped. This is exemplified by the fact that the "Other" category is too large. Finally, the data provide evidence for both a waterfall view and an interactive, cyclic view of software development. We conclude that the prototyping effort is a necessary part of developing and installing any large-scale process monitoring experiment.>