An in-depth look at approximation via deep and narrow neural networks

In 2017, Hanin and Sellke showed that the class of arbitrarily deep, real-valued, feed-forward and ReLU-activated networks of width forms a dense subset of the space of continuous functions on , with respect to the topology of uniform convergence on compact sets, if and only if holds. To show the necessity, a concrete counterexample function was used. In this note we actually approximate this very by neural networks in the two cases and around the aforementioned threshold. We study how the approximation quality behaves if we vary the depth and what effect (spoiler alert: dying neurons) cause that behavior.