Embedding for Informative Missingness: Deep Learning With Incomplete Data

Deep learning is increasingly used to make pre-dictions on biomedical and social science data. A ubiquitous challenge in such applications is that the training data is often incomplete: certain attributes of samples could be missing. Moreover, there could be complex structures in the pattern of which attributes are missing - for example, whether the glucose level is measured for a participant may depend on his/her other attributes (e.g., age) as well as on the prediction target (say, diabetes status). We propose a general embedding approach to learn representations for missingness. The embedding can be a modular layer of any neural network architecture and it's learned at the same time as the networks learn to make predictions. This approach bypasses the need to first impute the missing attributes, which is a key limitation because standard imputation methods require random missingness. Our systematic experimental evaluations demonstrate that missingness embedding significantly improves the prediction accuracy especially when the data missingness has structures, which is typical in practice. We show that the embedding is robust to changes in the missingness of test data (domain-adaptation) and discuss how the embedding reveals insights on the underlying missing mechanism.