Innovation capacity of dynamical learning systems

In noisy physical reservoirs, the classical information-processing capacity Cᵢₚ quantifies how well a linear readout can realize tasks measurable from the input history, yet Cᵢₚ can be far smaller than the observed rank of the readout covariance. We explain this ``missing capacity'' by introducing the innovation capacity Cᵢ, the total capacity allocated to readout components orthogonal to the input filtration (Doob innovations, including input-noise mixing). Using a basis-free Hilbert-space formulation of the predictable/innovation decomposition, we prove the conservation law Cᵢₚ+Cᵢ=rank(ΣXX)≤ d, so predictable and innovation capacities exactly partition the rank of the observable readout dimension covariance ΣXX∈ Rd× d. In linear-Gaussian Johnson-Nyquist regimes, ΣXX(T)=S+T N₀, the split becomes a generalized-eigenvalue shrinkage rule and gives an explicit monotone tradeoff between temperature and predictable capacity. Geometrically, in whitened coordinates the predictable and innovation components correspond to complementary covariance ellipsoids, making Cᵢ a trace-controlled innovation budget. A large Cᵢ forces a high-dimensional innovation subspace with a variance floor and under mild mixing and anti-concentration assumptions this yields extensive innovation-block differential entropy and exponentially many distinguishable histories. Finally, we give an information-theoretic lower bound showing that learning the induced innovation-block law in total variation requires a number of samples that scales with the effective innovation dimension, supporting the generative utility of noisy physical reservoirs.