How Many Labels Until We Know? Sequential Detection of Careless Annotators from Their Agreement with Others

Purpose. Human annotation is the behavioural data on which many AI systems learn what emotions, toxicity or relevance look like. When some annotators answer carelessly, labels and models degrade. Existing quality checks usually run after the task. We ask how quickly a careless annotator can be recognised from their agreement with others while the task is under way, and what this does to label quality. Design and methods. We used the GoEmotions raw data (211,225 annotations of 58,011 Reddit comments by 82 annotators; 76 with at least 100 scored annotations). Each annotation is scored as a hit if its sentiment-group vector (positive, negative, ambiguous, neutral) equals the majority of the other annotators of the same comment. A restarted sequential probability ratio test (CUSUM form) accumulates the log-likelihood ratio between a "careful" hit rate (0.342, the 10th percentile of real annotators) and a "careless" hit rate (0.2) and flags the annotator when it passes a threshold set by the error level α. We evaluated detection delay on 32 injected synthetic spammers per repetition (random, always neutral, sampled from the label prior, sticky), on real annotators whose behaviour was switched to careless after 300 annotations, and on the quality of the resulting data, and compared with fixed-sample binomial tests. Findings. All injected spammers were flagged. At α = 0.01 the median number of annotations needed was 48.5 for random, 50.5 for prior-sampled, 43 for sticky and 93.5 for always-neutral spammers, whose agreement with the majority (0.223) is close to the careless hypothesis. Fixed binomial tests flagged 70.8% of random spammers after 60 annotations and 95.8% after 120. Real annotators who became careless after 300 annotations were flagged after a median 49 further annotations, while 2.0% were flagged before the change. Injected spammers (18.5% of annotations) lowered the mean agreement of the dataset from 0.433 to 0.384; removing the annotators flagged at α = 0.01 restored it to 0.431. Among the real annotators, 5 were persistently flagged at α = 0.01 (and 1 at α = 0.001); they are the annotators with chronically low agreement, not necessarily careless ones. Originality. The study turns annotator quality control into an online decision problem with explicit error control, measures how many labels are needed to detect different kinds of careless behaviour, and shows with real annotators that the same detector catches mid-task behaviour changes while keeping early false flags low.