Foundations of multimodal representations: a taxonomy of representational modalities

Advances in information technologies are producing a very large number of possible interface modality combinations which are potentially useful for the expression and exchange of information in humancomputer interaction. However, a principled basis for analysing arbitrary input/output modality types and combinations as to their capabilities of information representation and exchange is still lacking. The paper presents a generative approach to the analysis of output modality types and their combinations and takes some steps towards its implementation, departing from a taxonomy of generic unimodal modalities of representation. A small number of key properties appear sufficient for creating a taxonomy of generic output modalities which is relatively simple, robust, intuitively plausible and reasonably complete. These (orthogonal) properties are: analogue and non-analogue representations; arbitrary and non-arbitrary representations; static and dynamic representations; linguistic and non-linguistic representations; different media of representation; and modality structure. The work presented is part of the larger research agenda of modality theory.