Time-based sampling: Theory and an application to video reconstruction from events

Traditional signal processing is based on the idea that an analogue waveform should be converted in digital form by recording its amplitude information at specific time instants. Nearly all data acquisition, processing and communication methods have progressed by relying on this fundamental sampling paradigm. Interestingly, we know that the brain operates differently and represents signals using networks of spiking neurons where the timing of the spikes encodes the signal's information. This form of processing by spikes is more efficient and is inspiring a new generation of event-based audio-visual sensing and processing architectures. In this talk, we investigate time encoding as an alternative method to classical sampling, and address the problem of reconstructing classes of sparse non-bandlimited signals from time-based samples. We consider a sampling mechanism based on first filtering the input, before obtaining the timing information using a time encoding machine. Leveraging specific properties of these filters, we derive sufficient conditions and propose novel algorithms for perfect reconstruction of classes of sparse signals. We then extend our analysis to multi-dimensional signals and introduce a deep neural networks for the reconstruction of intensity videos from events generated by event-driven cameras. The network is obtained by embedding prior-models in its architecture. We show that this approach leads to simple and yet effective neural networks.