What does this research mean for the field?

The HeuCI Usability Violation Dataset provides a comprehensive benchmark for evaluating automated usability heuristic evaluation tools in CI/CD pipelines. Novelty: ClaimNovelty.SYNTHESIS. Consensus alignment: ConsensusAlignment.NEUTRAL.

What question did this study set out to answer?

The goal is to create a comprehensive dataset for evaluating the effectiveness of usability heuristic evaluation tools in CI/CD pipelines.

February 28, 2026Open Access

HeuCI Usability Violation Dataset: Benchmark for Automated Heuristic Evaluation in CI/CD Pipelines

Key Points

The goal is to create a comprehensive dataset for evaluating the effectiveness of usability heuristic evaluation tools in CI/CD pipelines.
Compiled 5,997 annotated code samples from various sources.
Included synthetic violations, real-world violations, and negative samples.
Categorized annotations by violation type, severity, and heuristic category.
Dataset features 3,200 synthetic violations and 1,847 real-world violations.
Annotations provide insights into usability issues encountered in open-source projects.
Facilitates benchmarking of automated evaluation tools for enhanced usability detection.

Abstract

This dataset contains 5,997 annotated code samples (3,200 synthetic violations, 1,847 real-world violations from 15 open-source projects, and 950 negative samples) for benchmarking automated usability heuristic evaluation tools. Annotations include violation type, severity, and heuristic category.

Read Full Paperexternally

Bookmark

View Full Paper

Cite This Study

Nwasra et al. (Tue,) studied this question.

synapsesocial.com/papers/69a287b00a974eb0d3c038f9 https://doi.org/https://doi.org/10.5281/zenodo.18749927

Bookmark

View Full Paper