Do Neural Language Models Overcome Reporting Bias?

Mining commonsense knowledge from corpora suffers from reporting bias, over-representing the rare at the expense of the trivial We study to what extent pre-trained language models overcome this issue. We find that while their generalization capacity allows them to better estimate the plausibility of frequent but unspoken of actions, outcomes, and properties, they also tend to overestimate that of the very rare, amplifying the bias that already exists in their training corpus.