A class of stochastic procedures for the assessment of knowledge

The knowledge state of an individual with respect to a particular body of information is conceptualized as the set of all the questions that this individual is capable of solving. The goal of an assessment procedure is to identify, by a sequence of appropriately chosen questions, the individual's state among all possible ones. A deterministic procedure is conceivable, but not realistic, in that it does not account for possible inconsistencies in the observed responses. Such inconsistencies may arise from careless errors or lucky guesses from the subject, but may also be of a more fundamental character. A stochastic framework is developed here, in which an individual ‘state’ is formalized as a distribution on the set of all possible knowledge states. On each trial, the assessor has a likelihood function on the set of knowledge states which provides the basis for selecting the question to be asked on that trial. The response is assumed to depend on the individual ‘state’. The question asked and the response observed are used to update the likelihood function. Several examples of questioning rules and of updating rules are discussed, which lead to Markov processes. A central problem is to describe conditions ensuring that the latent distribution corresponding to the subject's ‘state’ can be estimated. We show that, under general conditions, the ‘state’ of a subject is uncoverable if the latent distribution is concentrated on a particular knowledge state.