Language models are big. But maybe size isn’t everything, according to Chris Edwards
A Thu, study studied this question.