Federal Statistical Confidentiality and Business Data: Twentieth Century Challenges and Continuing Issues

The roots of the modern concept of statistical confidentiality in the US federal statistical system can be traced directly back to the late nineteenth century efforts of statisticians to ensure full and accurate responses by businesses to statistical inquiries. Officials argued that such confidentiality guarantees were needed to ensure that the providers of enterprise and establishment data could be confident that the statistical agencies could not be forced to share their responses with others, such as regulatory or tax authorities, congressional investigators, prying journalists, and competitors, who might use this information to the detriment of the data provider. Nevertheless, over the years, the principle of statistical confidentiality with respect to information provided by businesses in statistical inquiries has been repeatedly challenged by other executive branch departments, independent regulatory agencies, the courts, Congress, and members of the public, with quite varied results. The paper uses the published record and archival research to examine the history of challenges to statistical confidentiality, and the responses of the statistical agencies, the federal statistical system as a whole, including the office of the chief statistician in OMB (and its predecessors), executive department and independent non-statistical agencies, the courts, and Congress as well as representatives of the business community. Long-term trends and the implications for maintaining and strengthening the confidentiality protections for establishment- and enterprise-level business data provided to federal agencies for statistical purposes are discussed.