Inference for Non-random Samples

Summary Observational data are often analysed as if they had resulted from a controlled study, and yet the tacit assumption of randomness can be crucial for the validity of inference. We take some simple statistical models and supplement them by adding a parameter θ which reflects the degree of non-randomness in the sample. For a randomized study θ is known to be 0. We examine the profile log-likelihood for θ and the sensitivity of inference to small non-zero values of θ. Particular models cover the analysis of survey data with item non-response, the paired comparison t-test and two group comparisons using observational data with covariates. Some practical examples are discussed. Allowing for sampling bias increases the uncertainty of estimation and weakens the significance of treatment effects, sometimes substantially so.