Integration themes in multimodal human-computer interaction

This research examines how people integrate spoken and written input during multimodal human-computer interaction. Three studies used a semi-automatic simulation technique to collect data on peoples free use of spoken and written input. Within-subject repeated-measures studies were designed, with data analyzed from 44 subjects and 240 tasks. The primary factors that govern peoples selection to write versus speak at given points during a human-computer exchange were evaluated. Analyses revealed that people write digits more often than textual content, and proper names more often than other text. A form-based presentation, in comparison with an unconstrained format, also increased the likelihood of writing. However, the most in#uential factor in patterning people s integrated use of speech and writing is contrastive functionality, or the use of spoken and written input in a contrastiveway to designate a shift in content or functionality, such as original versus correc...