What question did this study set out to answer?

This research aims to evaluate the effectiveness of LayerForge in enhancing AI output relevance for small-N text domains.

May 15, 2026Open Access

LayerForge: Deterministic Layer Decomposition for AI Output Convergence — An Empirical Report on Newman Modularity Outperforming CPM in Small-N Text Domains

Key Points

This research aims to evaluate the effectiveness of LayerForge in enhancing AI output relevance for small-N text domains.
Implemented a deterministic pipeline, LayerForge, utilizing community detection and thematic layer decomposition.
Conducted empirical comparisons of Newman modularity against the Constant Potts Model (CPM) in small-N settings (N=20-40).
Assessed performance through multiple metrics including dual-method correlation sweeps and ARI/NMI comparisons.
Newman backend achieved an exact match of 87.5%, significantly outperforming CPM with a match rate of 19%.
Achieved ARI scores of 0.430 for CPM and 0.557 for Newman when K=4 was applied.
Findings indicated strong effects favoring Newman, contrary to initial theoretical expectations that CPM would succeed.

Abstract

Conversational AI output tends to include content beyond what the questioner is actually concerned with. We report on LayerForge, a deterministic pipeline that decomposes a corpus into 4±1 thematic layers and returns only the layer (s) relevant to a query. The pipeline combines five established components — Cowan's working-memory upper bound, Newman modularity for community detection, SCA-style per-layer distillation, HERCULES-style hierarchical KMeans, and a deterministic core with no LLM in the analysis loop — chosen so that "AI noise filtering" does not itself depend on an AI. We re-implemented a CPM (Constant Potts Model) backend (MIT-pure, self-implemented) to test whether the theoretically resolution-limit-free alternative would outperform Newman in this small-N text domain (operational target N=20-40, external scaling check up to N=100 on 20 Newsgroups). Across 119-row dual-method correlation sweep, 196-cell N×K heatmap, 95-row ARI/NMI comparison, and the external 20-Newsgroups benchmark, the Newman backend exceeded CPM by a large effect size in our test range (Hₛtruct exact match 87. 5% vs 19%; 20NG ARI 0. 430 vs 0. 239 at default K=3, Newman reaches 0. 557 when K=4 is forced; same direction at N=100), against the naïve theoretical expectation that CPM should win. Positioning: LayerForge shares its core substrate (sentence-embedding similarity + community detection) with prior work in clustering-based topic modeling, notably G2T (Zhang et al. 2023). The contribution of this paper is twofold: (i) the implementation report of combining SCA decomposition, HERCULES hierarchical skeleton, the 4±1 cognitive constraint, and a Mode A/B/C operational layer, and (ii) the systematic empirical comparison of Newman vs CPM in the small-N text regime, including a three-failure-mode mechanism analysis. Concurrently with this work, Bohdal et al. (Samsung 2026) independently arrived at the same problem space on the LLM memory token side; we treat this convergence as evidence that the design space is being identified by multiple independent groups. We report an empirically derived method-selection rule (concept-only; runtime implementation deferred because in the studied domain the rule would output "newman" in essentially all cells). The artifact is published as a Claude Code skill (LayerForge itself does not hold an API key; LLM access at Boundary 1/2 is mediated through Claude Code), and is reproducible end-to-end.

Read Full Paperexternally

اسأل الذكاء الاصطناعي

Bookmark

View Full Paper