A practical guide from Ordinary Least Squares to causal machine learning

This article shows how standard econometric methods, such as Ordinary Least Squares (OLS), provide a valuable basis for understanding state-of-the-art Machine Learning. We introduce nonlinearity within a polynomial regression framework to illustrate the bias–variance trade-off and overfitting and then extend the discussion to regularization methods such as LASSO. The model-selection procedure, i.e., cross-validation, is broken down into manageable steps, each illustrated with visual aids, to clarify how Supervised Machine Learning predicts outcomes. Subsequent sections explain the limitations of these prediction methods and how they can be adapted for causal inference. We also highlight the potential and limitations of Machine Learning in variable selection for regression models to increase the replicability and credibility of empirical results and thus contribute to the p-value debate.