Optimization of color-coded apertures for compressive multispectral and depth imaging with time-of-flight sensors

The demand for new image sensors that can simultaneously acquire multispectral-and-depth imagery (MS-D) using compact, lightweight and monocular imaging systems is rapidly increasing. Seminal works in this line focused on RGB-D sensors that were able to acquire 3-channel color images and a depth map, but relying on two independent image sensors. We intend to advance the state-of-the-art in imaging systems with the ability to extract depth while accurately capturing spectral properties of scenes using as few as a single snapshot. Therefore, this paper discusses the advances of a compressive spectral-depth computational camera that employs a time-of-flight (ToF) sensor, an optimized color-coded aperture (CCA), a dispersive element and a model-based reconstruction algorithm, to attain MS-D imaging. In particular, the ToF sensor has the ability to measure ambient light along with modulated light from the active source, and the CCA is a passive-and-static optical element optimized to spectrally encode the scene reflectance while permitting the propagation of the active modulated light unaffected. The optimization of the CCA is performed using a direct-binary-search (DBS) algorithm that exploits the underlying ideas of blue-noise multitoning, to design the spatial distribution and spectral response characteristics of each one of the optical filters (pixels) of the CCA. We report a proof-of-concept prototype of such a camera that uses a CCA fabricated with 3 different filters using deposition and lithographic patterning cycles of thin-films.