The Distribution Is the Performance

ince the dawn of computing, the established practice for performance evaluation has been to represent performance as a variable for some metric and measure or estimate it as accurately as possible.However, computer hardware and software have evolved in ways that render their performance a constantly varying value, which means that any single performance summary may no longer be representative of the system.Instead, performance is now often behaving like a complex random variable, and should therefore be handled as such, with the appropriate statistical tools.Obviously, there is nothing novel about the requirement to handle complex distributions with appropriate statistical tools.For example, biologists, psychologists, clinical researchers, and social scientists have been managing uncertainty in their experiments and observations for decades.What's changed is that computer performance has evolved from quantities that were relatively easy to measure, model, and reproduce, to complex random variables with ostensibly nondeterministic or irreproducible behaviors.In this column, we review the confluence of recent factors that are changing performance evaluation, illustrate why the "old ways" of performance evaluation are no longer good enough, describe the evolution of performance evaluation in reaction to these changes, and predict some possible directions for the future of performance evaluation.