An Empirical Study on the Selection of Good Metamorphic Relations

Full software test automation requires automated test input generation, execution, and output evaluation. The latter task is non-trivial and usually referred to as the oracle problem in software testing. The present paper describes an empirical study on metamorphic testing, an approach to the oracle problem. This study was conducted with common Java implementations of determinant computation in order to evaluate the usefulness of the metamorphic testing approach and to establish general criteria that can be used to quickly assess metamorphic relations with respect to their suitability. The latter is very important, since metamorphic testing is based on so-called metamorphic relations on input-output tuples, which can easily be found. It is, however, crucial to evaluate these relations according to their usefulness. The empirical study enables us to derive general rules that can be used to quickly assess metamorphic relations and identify those that should be considered and studied in more detail with other methods (e.g. with mutation analysis)