What question did this study set out to answer?

Explore the role of user interfaces (UIs) in mediating human and AI interactions across various modalities.

April 17, 2026Open Access

Human-AI-UI Interactions Across Modalities

Key Points

Explore the role of user interfaces (UIs) in mediating human and AI interactions across various modalities.
Conduct workshop discussions on UI design for intelligent agents
Examine the impacts of multimodal inputs on UI responsiveness
Assess challenges in transparency and accessibility in XR environments
Identified strategies for achieving alignment between human intents and AI actions
Highlighted the transformative potential of generative models in UI development
Discussed the challenges around user agency and accuracy in multitasking environments

Abstract

Designing and developing user-friendly interfaces has long been a cornerstone of HCI research, yet today we are at a turning point where UIs are no longer designed solely for humans but also for intelligent agents that act on users' behalf, while UIs are also expanding beyond 2D screens into extended reality environments with inherently multimodal characteristics, together challenging us to rethink the role of the UI as a mediator of human-AI interaction.This workshop will explore how UI agents bridge human intent and system behavior by interpreting multimodal inputs and generating adaptive outputs across surfaces from screens to extended reality (XR), and we will examine not only their technical capabilities but also their broader impact, including how agents reshape daily workflows, how bidirectional alignment between human and AI activity can be achieved, and how generative models may transform UI creation.XR provides a compelling testbed for these questions and highlights challenges around accuracy, efficiency, transparency, accessibility, and user agency, setting the stage for the next generation of intelligent and adaptive UIs.

Bookmark

View Full Paper