Topics in structured nonparametric regression: uncoupled isotonic regression and tree-based learning

This thesis brings together several contributions to nonparametric regression, ranging from isotonic regression with uncoupled data to the analysis of tree-based methods, from a theoretical and applied viewpoint. On the one hand, we consider the estimation of an isotonic regression function in a setting where the pairing between observed responses and design points is unavailable to the statistician. This makes the problem significantly different from standard regression problems where observations come as pairs. Under the assumption that the noise variance decays to zero, we consider two estimators that rely on distributional properties of the observed responses, without assuming knowledge of the noise distribution. We derive consistency rates and asymptotic normality under smoothness assumptions. Furthermore, we identify a phase transition property driven by the decay rate of the noise variance: the estimation problem transitions from being harder than the standard regression problem to being equally hard. On the other hand, in the standard regression model with data available as pairs, we consider tree-based estimation using Random Forests and variants of it. A general consistency result valid for a large class of tree-based algorithms is established, by generalizing a recently introduced sufficient impurity decrease condition. Furthermore, we investigate drawbacks of Random Forests in case of pure interactions, that is, when there are interactions between two or more covariates without the corresponding main effects. In a large simulation study, we observe that Random Forests perform poorly in such settings and we show that simple adaptions of the tree-building procedure - one of which is novel - improves the performance in such settings while not sacrificing predictive power in other scenarios. In addition, we prove that the adaptions are consistently estimating regression functions with pure interactions by employing our general theory. Finally, we also observe differences on real data, with the interaction-specific methods outperforming Random Forests in some examples, and performing comparably in others.