NeuroEval: Evaluating Data Transparency, Reproducibility, and Neural-Coding Rigor in Computational-Neuroscience Simulation Papers

This deposition bundles the NeuroEval Reproducibility Report — an arXiv-style preprint that explains how the NeuroEval evaluation system works and presents its evaluations of a corpus of computational-neuroscience simulation papers — together with the complete, MIT-licensed source code of the system. Two files are included: the report PDF and a full archive of the source at the corresponding revision. Computational-neuroscience papers increasingly rest on neural simulators — NEURON and NetPyNE, NEST, Brian2, The Virtual Brain, Nengo, PyNN, and spiking-network libraries — yet the reproducibility of the models they describe is rarely assessed systematically. NeuroEval is an LLM-driven framework that submits a paper by URL or PDF, extracts its real text (and errors out rather than scoring unextractable input), gathers external evidence, and scores the paper on a versioned seven-dimension rubric using Anthropic Claude. Six dimensions measure conformance to good data and reproducibility practice and together make up the Reproducibility score: Data & Stimulus Disclosure (15%), Dataset/Model Resolvability (12%), Simulator & Code Availability (12%), Model-to-Code Traceability (15%), Neural Model Spec Clarity (14%), and Reproducibility Package Quality (7%). Identifiers are resolved live against the neuroscience standards ecosystem — ModelDB, NeuroML-DB, Open Source Brain, the NeuroML/SONATA/PyNN toolchains, DANDI, and BioModels, plus repository facts and a Crossref/DataCite DOI resolver — rather than trusting the paper's own claims. A seventh, orthogonal dimension, Neural Information Processing / Representation-Geometry Rigor (25%), scores whether a paper formalizes and quantifies how the system encodes or transforms information. As of rubric 0.2.0 this admits both classical information theory (spike-train entropy, mutual and transfer information, channel capacity, bits per spike, coding efficiency) and population/representation geometry (Fisher information, representational dissimilarity, decoder discriminability, neural-manifold dimensionality and capacity) — but geometry counts only when it is tied to discriminability, decoding, or behavior; a bare PCA/UMAP/t-SNE projection earns no credit. It measures scientific-content rigor, not transparency; infrastructure and simulator-tool papers that make no information-processing claim are scored at a neutral baseline (~50 on the 0–100 scale), while simulator-support work can exceed the midpoint when it enables reproducible representation or information analysis. It carries the single largest weight of the seven dimensions. The overall score is the weighted mean of the seven dimensions. The bundled report applies the rubric (version 0.2.0) to a corpus of computational-neuroscience projects, reporting results as a split: a Reproducibility score (the six transparency dimensions, renormalized to 100), the orthogonal information-processing axis on its own, and the combined seven-dimension Overall. Its headline finding is that transparency is uneven and reproducibility packaging consistently weak, while quantified neural-information-processing rigor remains the exception even across projects whose modeled phenomenon is information processing itself, with infrastructure and tool papers correctly resting at the neutral baseline. This is report edition 0.5.1, a pre-1.0 release; the evaluation rubric is version 0.2.0. NeuroEval shares its evaluation engine with its sibling tool BioEval but carries a computational-neuroscience rubric and registry. Built as a pnpm monorepo: React + Vite frontend, Express 5 API, PostgreSQL + Drizzle ORM, contract-first OpenAPI/Orval codegen.