Researcher judgment and study design: Challenges of using administrative data

BACKGROUND: Questions have been raised about methods of studies finding substantial undercounting of workplace injuries and illnesses by the Bureau of Labor Statistics (BLS) and workers' compensation agencies. A more recent study of Minnesota concluded that the BLS survey captures 84-90% of workers' compensation cases. METHODS: We examined the sensitivity of findings in two studies to alternate sample definitions and study assumptions. RESULTS: Applying alternate sample construction rules to the earlier study increased estimated BLS reporting rates from 68% to 77%, assuming source independence. Applying alternate assumptions to the more recent Minnesota study reduced its high estimate of BLS reporting from 90% to 53-64%. CONCLUSIONS: Studies linking administrative data from different sources require substantial judgment in constructing research datasets and choosing analytic methods. Moreover, different sample construction rules lead to different results. This suggests that sensitivity analysis should be carried out when alternatives cannot be ruled out. In this case, sensitivity analysis supports the hypothesis of substantial underreporting.