Exact prevalence regions: finite-sample uncertainty quantification, threshold decisions, and monitoring for class-prevalence analytics under prior probability shift

Many analytics decisions consume class prevalences rather than individual labels: the market share of a sentiment, the case burden of a disease, the drift state of a deployed model. Quantification methods estimate the prevalence vector of an unlabeled batch under prior probability shift, but they return point estimates, and their uncertainty companions, resting on large-sample approximations, resampling, or a prior, carry no finite-sample frequentist guarantee. I introduce exact prevalence regions (EPR): confidence regions obtained by inverting exact Monte Carlo two-sample tests against reference samples drawn from held-out, class-conditional calibration pools. Because prior probability shift makes each null hypothesis exactly simulable, the region is finite-sample valid for any classifier, any permutation-symmetric statistic, and any sample size, marginally over calibration sampling; the exchangeability the test uses is derived from the sampling assumptions, and the cost of its failure under within-batch dependence is measured. The same construction yields exact one-sided tests for threshold decisions and a day-over-day monitoring workflow whose representability alarm flags batches that leave the label-shift family. Across 18 datasets and more than 300,000 evaluation bags, against audited bootstrap, delta-method, and Bayesian baselines, EPR held its nominal level wherever the shift assumption held by construction, including 0.950 on all 5000 official bags of the 28-class LeQua 2022 task, where bootstrap coverage spanned 0.030–0.974 across implementations and seeds. The cost is width, 1.13–1.43 times the delta method’s outside rare-class regimes, and reduced power in threshold decisions; both are quantified and distilled into practitioner guidance.