Learning neural representations for 2d and 3d reconstruction

Two-dimension (2D) and three-dimension (3D) reconstruction are foundational tasks in computer vision that underlie various downstream analyses, with applications ranging from medical imaging and computer graphics to video comprehension. Advances in computational power and deep learning have led to rapid progress in these tasks, motivating the need to properly evaluate which model architectures are best suited for different reconstruction tasks and application domains. In this thesis, we will explore various problems in 2D and 3D reconstruction and the novel methods that can be utilized to solve them. In the first chapter, we examine implicit neural representations (INR) used for novel view synthesis and analyze how the choice of INR influences reconstruction quality in NeRF-based architectures. In the second chapter, we explore 2D and 3D reconstruction methods for medical imaging data, focusing particularly on evaluating how effectively generative methods such as Variational Auto-Encoders (VAE) and U-Nets can reconstruct undersampled or noisy medical images. In the final chapter, we address key challenges in 3D reconstruction: pose estimation, feature field reconstruction, and sparsity. We begin with an assessment of the utility of emerging 3D foundational models, such as VGGT, Map-Anything, and MAST3R, in reconstructing accurate 3D poses. We then evaluate the efficacy of models such as LeRF and Feature Splatting at learning feature fields that can be used to localize objects in a set of egocentric data. Lastly, we finish with a discussion on sparsity and how geometric priors can be used to overcome limited amounts of data. By evaluating these methods on a series of tasks, we seek to evaluate which types of methods perform the best on their respective task and uncover broader trends in visual reconstruction that can be used to inform future work.