Likelihood-Free Estimation of IRT Models in Small Samples: A Neural Networks Approach

Existing estimators of parameters of item response theory (IRT) models exploit the likelihood function. In small samples, however, the IRT likelihood oftentimes contains little informative value, potentially resulting in biased and/or unstable parameter estimates and large standard errors. To facilitate small-sample IRT estimation, we introduce a novel approach for small-sample IRT estimation that does not rely on the likelihood. Our estimation approach capitalizes on item pool information and trains a neural network (NN) to interpolate the relationship between responses and item parameters. We describe and evaluate our approach for the three-parameter logistic (3PL) model; however, it is applicable to any model with an item characteristic curve (ICC). Three types of NNs are developed, supporting to obtain both point estimates and confidence intervals for IRT model parameters. The results of a simulation study demonstrated that these NNs perform better than Bayesian estimation using Markov chain Monte Carlo (MCMC) methods in terms of quality of the point estimates and confidence intervals while also being much faster. Among others, these properties facilitate (1) to pretest items in a real-time testing environment (e.g., CAT), (2) to pretest more items and, as a consequence, to replenish operational pools quicker and assemble more tests as well as (3) to pretest items only in a secured environment (e.g., in strictly proctored test centers) to eradicate possible compromise of new items in online testing.