What question did this study set out to answer?

This research aims to evaluate the impact of dense connectivity on autoregressive language modeling performance.

March 14, 2026Applied Sciences0 citationsOpen Access

Beyond Perplexity: A Multi-Faceted Analysis of a Novel Densely Connected Transformer

ESEnrico De SantisSapienza University of Rome AMAlessio MartinoLibera Università Internazionale degli Studi Sociali Guido Carli ARAntonello RizziSapienza University of Rome

Key Points

This research aims to evaluate the impact of dense connectivity on autoregressive language modeling performance.
Compared standard Transformer decoder with dense decoder on Penn Treebank and WikiText-2.
Conducted tests under two different fairness regimes concerning training conditions and parameter budgets.
Analyzed results focusing on test perplexity and structural differences between model outputs.
Dense connectivity did not lead to a consistent reduction in test perplexity across datasets.
On WikiText-2, the standard model performed better in both testing regimes.
Improvements in perplexity on Penn Treebank were minimal and dependent on the testing conditions.

Abstract

Background: Dense cross-layer connectivity can shorten gradient paths and promote feature reuse, potentially improving optimization under fixed training budgets. Objective: We test whether concatenation-based dense historical connectivity improves decoder-only autoregressive language modeling under controlled comparison protocols. Methods: We compare a standard Transformer decoder and a dense decoder on Penn Treebank and WikiText-2 under two fairness regimes: (i) a same training recipe setting with a fixed baseline and a bounded dense architectural search, and (ii) a same parameter budget setting where the dense model is resized to not exceed the baseline parameter count. Results: Dense connectivity does not consistently reduce test perplexity; on WikiText-2, the baseline remains better in both regimes, while gains on Penn Treebank are small and regime-dependent. Ablations within the dense family show that depth and feed-forward capacity are the most reliable drivers of perplexity improvements. Conclusions: Probes and attention diagnostics do not reveal a clear advantage for dense connectivity in our limited probe set, while Zipf–RQA analysis of long-form generations reveals systematic structural differences between baseline and dense outputs. Specifically, Zipf–RQA is used here as a descriptive structural probe rather than a performance metric.

Ask AI

Helpful

Bookmark

View Full Paper