A comparative analysis of single-page and site-wide tools with the OPTIMAL-EM framework

Web accessibility auditing is critical for ensuring websites are accessible to those with disabilities, and whilst automated tools cannot evaluate every accessibility barrier, they can help evaluate specific barriers and illustrate the accessibility of a page. This paper proposes applying our OPTIMAL-EM framework for evaluating web accessibility evaluation tools, comprising the following six metrics to conduct a comparative analysis of single-page and site-wide accessibility evaluation tools: coverage, complexity, accessibility, representativeness, popularity and freshness. Our comparative analysis identified gaps in the current tooling regarding popularity and representativeness and recommends incorporating a method of prioritising web pages and support for representative sampling to make web accessibility evaluation tools more effective and efficient. Specifically, we identify the absence of representativeness measures in the current suite of accessibility tools, a lack of support for regional accessibility standards, and the potential use of popularity measures to optimise the web accessibility evaluation process. The main contribution of this paper is to demonstrate how evaluation tools can be systematically assessed with our OPTIMAL-EM framework.