Empirical Bayes for Learning to Learn

We present a new model for studying multitask learning, linking theoretical results to practical simulations. In our model all tasks are combined in a single feedforward neural network. Learning is implemented in a Bayesian fashion. In this Bayesian framework the hidden-to-output weights, being specific to each task, play the role of model parameters. The input-to-hidden weights, which are shared between all tasks, are treated as hyperparameters. Other hyperparameters describe error variance and correlations and priors for the model parameters. An important feature of our model is that the probability of these hyperparameters given the data can be computed explicitely and only depends on a set of sufficient statistics. None of these statistics scales with the number of tasks or patterns, which makes empirical Bayes for multitask learning a relatively straightforward optimization problem. Simulations on real-world data sets on single-copy newspaper and magazine sal...