Usability testing is critical to developing equitable medical devices and digital health tools; however, traditional methods have been found to be resource-intensive and inconsistent, posing concerns for the adequate representation of marginalized communities in medical solution development. Artificial intelligence (AI)-driven usability testing methods have emerged as a promising solution to assist user experience (UX) analysts with their evaluation and mitigating potential evaluator bias. However, its reliability, particularly its impact on inclusivity and equity of marginalized communities, remains uncertain. Following the findings of a previous narrative literature review, this competitive evaluation assessed on-the-market AI-informed tools from seven prominent usability testing platforms, comparing the current state of AI-driven usability testing in the literature and commercially, using an adapted Society of Automotive Engineers five levels of automation and a three-level equity consideration scale. Six platforms offered Level 1 automation AI-products, assisting UX evaluators with facilitation and data analysis, while one achieved Level 3 conditional automation. Four platforms did not explicitly address the equity impact of their products, with only one platform incorporating a bias-reduction feature. Overall, AI-informed tools provide potentially inexpensive usability testing alternatives for digital health tools; however, more research is required to validate the consistency, accuracy, and reliability of these tools in usability testing practice.
Lombardi et al. (2025) studied this question.